Environmental summaries, drafted in Teams

The problem

When a large development needs planning permission, an Environmental Impact Assessment produces an Environmental Statement: hundreds of pages split into chapters such as air quality, noise, ecology and transport. Each chapter, often 1,000 to 20,000 tokens long, needs a Non-Technical Summary that the public can read.

A specialist wrote each summary by hand. The firm already had years of past chapters and the summaries written for them. That archive was the opportunity.

The approach

This is a style task more than a question-answering task. Fine-tuning would have cost time and money, so I started with retrieval: find the past chapters most like the new one, show the model those chapters next to the summaries people wrote for them, and ask it to write the new summary the same way.

1. Upload in Teams. A consultant drops a PDF or Word chapter into a chat with the bot.
2. Acknowledge. Azure Bot Service replies at once; an Azure Function takes the work.
3. Read the document. Azure Document Intelligence turns the PDF, tables included, into Markdown.
4. Clean. Strip headers, footers and figures; merge split tables; repair lists.
5. Classify. Decide the chapter type: air quality, noise, ecology and so on.
6. Retrieve. Find the most similar past chapters of that type and fetch their summaries.
7. Write. Azure OpenAI drafts the summary, using the retrieved pairs as worked examples.
8. Reply. The bot sends the draft back to the same chat.

The key decision: why TF-IDF beat embeddings here

My first design used dense embeddings. Long chapters have to be split into many chunks, and averaging their embeddings made chapters on different subjects look alike. A PCA plot confirmed it: topics barely separated.

So I built one TF-IDF index per chapter type. Training each index only on its own type means the terms that matter within that type (a pollutant, a receptor, a road name) stand out instead of being drowned by words every chapter uses. Matches were sharper, and anyone could see why a past chapter was chosen.

The lesson: test retrieval on your own documents before choosing the method everyone uses.

Engineering

• Asynchronous by design. Chapters take minutes to process, so the bot replies at once and the Function sends the result as a proactive message.
• Sized to the job. About twenty users and occasional use: consumption-plan Functions and a free-tier bot, with no queue to maintain.
• A catalogue, not a vector store. Cosmos DB links each past chapter to its summary and holds the metadata; Blob Storage holds the Markdown and the trained indexes.
• Clean data first. Most of the quality came from preprocessing. Where splitting chapters automatically was unreliable, I labelled them by hand with the experts.
• Observable. Timing for each stage and errors go to Application Insights.

What I’d add next

An evaluation set: a held-out group of past chapters, scored against their human summaries for coverage of key findings and readability, run before each change. Then a reviewer view that shows which source passages support each sentence of the draft.