The team's Slack bot could answer from Jira and Bitbucket. Every spec, runbook and standard kept in Google Drive was invisible to it.
- Client
- An agency with several in-house development teams
- Users
- Developers who ask a Slack assistant about tickets, estimates, bugs and internal documents
- Engagement
- Google Drive search with cited answers, plus hardening the existing assistant to run as a service
The outcome in six lines
| Measure | Before | After | Evidence |
|---|---|---|---|
| Where the assistant can find an answer1 | Jira tickets and Bitbucket commit summaries only | Google Drive as well, in 7 file types, with the source files linked | Measured |
| Automated tests2 | none | 226, run inside the built image on every development and UAT build, apart from 8 live tests run on demand | Measured |
| One Slack event delivered to many threads at once3 | two threads could both accept the same event | 50 threads racing on one event, exactly one accepted | Measured |
| Time logged on epics with sub-tasks4 | would have shown as empty for 50 of the 67 epics with logged work | sub-task time read from the aggregate field and shown, with tests over the whole chain | Measured |
| Document search under load7 | no document search | in a test setup of 25 synthetic documents, 187 ms at the 95th percentile over 100 searches, 20 at a time; one full answer in 5.9 s | Measured |
| How close an excerpt must be to reach the model6 | no document search | cutoff 0.70 in cosine distance; in a live test relevant excerpts scored 0.21 to 0.44, unrelated ones 0.84 to 0.97 | Measured |
Each line carries a counted or timed figure. The small number beside each line says how it is known.
The problem, in the client's terms
Our client's development teams already had a Slack assistant, built before we joined. It handled five kinds of request, greetings, creating a Jira ticket, looking up a ticket, estimating new work and suggesting a fix for a bug. For the last two it searched a nightly copy of Jira tickets and Bitbucket commit summaries by vector similarity and asked a language model to answer with those examples in hand.
Much of what the teams know is in neither place. Requirements, specs, runbooks and coding standards live in Google Drive as Docs, Sheets, Slides, PDFs and Office files. The assistant could not see any of them, so a question like "what do our docs say about authentication?" had no good answer.
So those questions went to whoever remembered where the document was, and the person who wrote the runbook kept answering for it.
The assistant also had operational gaps. It ran on a development web server, opened a new database connection per query, shared in-memory state across threads without a lock, had internal endpoints with no access control of their own, and could leave a user looking at a frozen "Analyzing your request..." message when the model was briefly unavailable. The client's tracker held both kinds of work, the reliability and security fixes and a story to add Google Drive.
- ConstraintDrive access is read-only, and the assistant sees only the folders a person chooses to share with it, the same way they would share with a colleague.
- ConstraintThe assistant reads Jira through one shared account, so it cannot apply each asker's permissions. Comments with restricted visibility stay out of every answer.
- ConstraintThe new front-door controls had to work without new infrastructure.
- ConstraintThe scheduled caller that starts the nightly sync gives up after 5 seconds.
- ConstraintNothing is created in Jira until the user types "create".
- ConstraintAnyone who can message the assistant can get excerpts from any folder shared with it, so what is shared is what is answerable.
The architecture, before and after
Switch between the two. The assistant kept its place between Slack and the team's tools. Google Drive joined its knowledge base, checks moved in front of every model call, and the nightly trigger was rebuilt so it no longer makes its caller wait.
- What hurt in this state
- Built in this engagement
Three decisions, and the options they rule out
Connecting a bot to a document store is the easy part. In a retrieval-augmented (RAG) assistant, these choices decide whether an answer can be trusted and whether the service stays up. Each one names what it replaced and what it cost.
What the search runs on
Turned down
One vector per whole document
A long document exceeds the embedding model's input limit, and one vector for a whole spec blurs what any part of it says.
Turned down
A separate vector database service
The assistant already kept its ticket and commit vectors in its relational database. A second store would mean another service to run and another copy of the data to keep in step.
Chosen
Chunks in the database the assistant already used
Text is split on line boundaries into chunks of up to 2,000 characters with about 200 of overlap, each embedded with its file name. A document and its chunks are written in one transaction, and both vector columns get a nearest-neighbour index for cosine distance.
What it cost: A document keeps at most 200 chunks, so the tail of a very long file is not searched, and one database now carries both the assistant's records and its search load.
How a document answer avoids inventing one
Turned down
Always hand the model the top results
The nearest chunks are returned even when nothing relevant exists, and a model given weak excerpts writes a confident answer from them.
Chosen
A distance cutoff, answers only from the excerpts, files cited
Chunks further than 0.70 in cosine distance are dropped. In a live test, relevant excerpts scored 0.21 to 0.44 and unrelated ones 0.84 to 0.97. The model answers only from what is left, names the document behind each fact and links it. If nothing passes, the assistant says it found nothing, without calling the model.
What it cost: A document worded very differently from the question can fall past the cutoff and be missed, and a cutoff set from one test run has to be checked again as the shared folders grow.
Where the duplicate and rate-limit state lives
Turned down
A shared cache service in front of the assistant
The controls had to work without new infrastructure, and a new service would need its own hosting, secrets and failure handling before the first fix shipped.
Chosen
In memory, under one lock, in one process with four threads
The duplicate check, the 3-second cooldown and the 15-a-minute budget run as one locked step, so two threads cannot both accept a Slack retry. Old records are pruned every 100 events or 120 seconds.
What it cost: The service cannot run a second worker until that state moves to a shared store. The limit is written into the container file and the code comments, so nobody scales it by accident.
How it was delivered
We joined the team that runs the assistant. Our software engineers did the hardening, the Drive work and the tests, alongside the client's own developers.
Keeping the prototype answering
A fix to the Jira search call and a cleanup for the memory that tracks Slack events.
Making it a service
Locks around shared state, a database connection pool, a production web server in place of the development one, and one distance metric across every vector search.
Fitting Slack's limits
Long answers split across message blocks, and Slack event de-duplication made atomic.
The front door and the tests
Ingress checks, a shared-secret key on internal endpoints, secrets checked at start-up, and the tests running in the build pipeline.
Better answers in the flows that existed
Jira comments and sub-task time in ticket answers, estimates that show their tickets, model retries with backoff, and cosine indexes.
Google Drive search
Drive ingestion, the document-search intent, Drive context in estimates and bug answers, then the sync fixes.
Who did the work
- Atyantik software engineersDrive ingestion and search, the two Drive prompts, the hardening work, the sync fixes and the 226 tests
- Client's teamThe original assistant, its hosting, and which Drive folders are shared with it
Who owns what now
The reliability and security work is on the client's UAT branch. Drive ingestion and document search are built and on the development branch, and the background sync trigger is finished on its own branch. Next would come per-user filtering of Drive results from the permissions already stored with each file, text recognition for the scanned PDFs the sync already flags, and a shared store for event and rate-limit state so the service can run more than one worker.
Results
What changes for the people using the assistant once the handed-over work ships, then two measurements that shaped how it was built.
- Developers asking in Slack on the development branch, get answers from specs, runbooks and standards with the files linked, or a plain "couldn't find anything".
- Whoever estimates new work in the UAT build, sees the past tickets an estimate was based on and the related documents, and an odd model reply such as "~6" is read as a number instead of crashing the answer.
- Leads reading an epic in the UAT build, see the time logged across its sub-tasks instead of an empty field.
- Anyone waiting on an answer in the UAT build, gets a try-again message when the model is briefly unavailable, instead of a placeholder that never changes.
No figure is given yet for how many questions the assistant answers from Drive, how often those answers helped, or how much time they save finding documents.
5 s
Caller's timeout
40.6 s
Drive sync of 25 documents
The fix starts the sync in the background, answers at once and refuses a second run while one is going. It is finished on its own branch and not yet merged.
| Option | Seconds. The scheduled caller gives up after 5 seconds. Syncing 25 documents from Drive alone took 40.6 seconds in testing, without Jira and Bitbucket. |
|---|---|
| Caller's timeout | 5 s |
| Drive sync of 25 documents | 40.6 s |
Source: Source note 5
0.44
Relevant excerpts, furthest seen
0.70
Cutoff
0.84
Unrelated excerpts, closest seen
Every relevant excerpt in the test was kept and every unrelated one was dropped, so the model saw only the excerpts that answered the question.
| Option | Cosine distance between a question and an excerpt, lower is closer. Excerpts further than the cutoff are dropped before the model sees them. |
|---|---|
| Relevant excerpts, furthest seen | 0.44 |
| Cutoff | 0.70 |
| Unrelated excerpts, closest seen | 0.84 |
Source: Source note 6
What looking for documents costs your team today
We do not publish what an engagement costs. So this works from your side: the time your team spends finding a document or asking a colleague where it is, the work a document-aware assistant can take on.
Cost of finding answers by hand, per year
$58,500
- Hours spent finding answers, per year
- 1,300 h
What went wrong, and what we would do first
The hosted model was retired, and every call failed
The model provider retired the model version the assistant was configured for, and every model call failed until a configuration change pointed it at a current version. No code changed. Next time we would add a start-up check that makes one small model call, and watch the provider's retirement notices for the version in use, so a retirement shows up as a warning before it shows up as an outage.
Logged time was lost four layers deep
Jira keeps sub-task time in a separate aggregate field, and four separate field lists in the code dropped it, so 50 of the 67 epics with logged work would have shown nothing. Each layer looked correct on its own. Tests now cover the whole chain. Next time we would test a ticket answer end to end against a real epic with sub-tasks before touching any single layer.
Will a document-aware Slack assistant work for your team?
It fits when
- Your team already works in Slack and keeps specs, runbooks and standards in Google Drive.
- The same "where is the doc for this" questions keep coming back to the same few people.
- Shared folders are a fair boundary for what the assistant may read.
Look elsewhere when
- Each asker must see only the documents they personally can open. Per-user permission filtering has to be built first, and it was not part of this engagement.
- Most of your documents are scanned PDFs. Text recognition comes before search, or the assistant finds file names and little else.
- Your documents cannot be sent to a hosted model provider. A self-hosted model changes the trade-offs, and the cost and quality picture should be checked first.
Check it yourself this week
If most of last week's questions were answered by a link to a document, an assistant could answer them from the document itself. Put your numbers into the calculator above.
- Search your team's Slack channels for last week's questions that start with "where" or "does anyone know".
- Count how many were answered with a link to a Google Drive file.
- Open three of those files and check they hold real text, not scanned images.
How it is built, by layer
- Chat surfaceSlack events, with the answer replacing a placeholder in the same thread
- Knowledge sourcesJira tickets and comments, Bitbucket commit summaries, and Google Drive through a read-only service account, refreshed nightly; a file removed or unshared drops out of search after the next sync
- Language modelA hosted large language model for intent, summaries and answers, with retries and backoff
- EmbeddingsA hosted text-embedding model, the same one for questions, chunks and summaries; each night only new or changed files are summarised and embedded again
- SearchVector columns with nearest-neighbour cosine indexes, in the relational database the assistant already used
- Document parsingGoogle files exported as text or spreadsheets, open-source parsers for PDF, Word, Excel and PowerPoint
- PromptsStored as database rows, added by migrations with rollback files, with token use logged per prompt
- ServiceOne web process with four threads, in a container on a managed cloud
- DeliveryThe client's build pipeline runs the tests inside the built image before it is pushed
How each figure was measured
- File types the Drive sync can read, counted from its export rules and parsers.
- Automated tests in the repository, counted before the work and after, run on every development and UAT build.
- A concurrency test that delivers one Slack event from 50 threads at once and counts how many are accepted.
- Epics in the client's Jira checked for time logged on their sub-tasks, before the field fix.
- The scheduled caller's timeout, and the time to sync 25 documents from Drive alone in testing.
- Cosine distance of relevant and unrelated excerpts in a live test run against shared documents.
- A test setup of 25 synthetic documents in a local database, timed over 100 searches at 20 at a time, and one document answer end to end.
Send us three questions your team asked in Slack last week, and we will tell you where an assistant would find the answers.
Tell us where your documents live and how your team asks for them. A software engineer reads it and replies with an honest answer, including when better folder structure would do.