Data annotation outsourcing

How data annotation outsourcing works in 10 steps

Data annotation outsourcing works best as a short procurement project: you define the work, test two or three providers on the same data, then scale the one that passes. The steps below follow that order.

1. 1

   Plan

   Write a project brief before you contact anyone

   • Takes: about an hour
   • Prevents: quotes you can't compare

   A project brief is a one-page description of the annotation work: what the data is, what each label must capture, how much there is and when you need it. Providers price from the brief, so a vague brief gets vague quotes.

   • Data type and file format (JPEG, MP4, WAV, PCD, PDF, JSONL)
   • Task and label list, with the number of classes
   • Volume in a clear unit: images, objects, video minutes, audio hours, documents or preference pairs
   • Quality target and how you will measure it
   • Deadline and delivery rhythm (one drop or weekly batches)
   • Security limits: personal data, health data, where data may be processed
   • 20 to 50 sample items, including the hard ones

   Send your project brief, get matched quotes

   We don't label data and we are not an annotation vendor. We read each brief and introduce annotation providers that work with that data type and task; they reply to you with questions or a quote. Your brief goes to those providers only if you tick the consent box.

   Your name

   Work email

   Company

   Data type Choose oneImageVideoTextAudioLiDAR / 3DOther or mixed

   Task Choose oneBounding boxesPolygons or segmentationKeypointsClassification or taggingVideo object trackingNamed entity recognition (NER)Sentiment or intentTranscription3D cuboids or point segmentationDocument fields or OCRRLHF, preference ranking or model evaluationRed-teamingOther

   Volume

   For example: 40,000 images, 300 audio hours

   Deadline Choose oneWithin 2 weeksWithin a month1 to 3 monthsMore than 3 monthsOngoing workNot fixed yet

   Short description

   What the model does, the label list, quality target, any security limits. Please don't paste personal data.

   Website

   I agree that dataannotationoutsourcing.com stores this brief and shares it with annotation providers it selects as a match, so they can contact me about this project. See the privacy notice.

   Send my brief

2. 2

   Match the data type to the annotation method

   • Takes: 1 to 2 hours
   • Prevents: paying for the wrong label

   The annotation method is the shape of each label: a tag, a box, an outline, a text span, a time-stamped transcript, a 3D cuboid or a ranking. Image annotation services, text annotation services, video annotation, audio annotation and LiDAR work are different jobs with different tools and skills, so name the method, not only the data type. "Boxes around pedestrians" and "pixel masks of pedestrians" are priced very differently.

   Data type	Common tasks	Output formats to request
   Image (computer vision labeling)	Classification, bounding boxes, polygons, semantic segmentation, instance segmentation, keypoints	COCO JSON, Pascal VOC XML, YOLO text files
   Video	Object tracking across frames with stable IDs, action and event tags, frame classification	Per-frame tracks in JSON or CSV
   Text (NLP data labeling)	Named entity recognition, sentiment analysis, intent annotation, classification, relations	CoNLL with BIO tags, JSONL spans
   Audio and speech	Transcription, speaker labels (who spoke when), sound events	WebVTT or SRT, time-stamped JSON
   LiDAR and 3D point clouds	3D cuboids, point segmentation, camera and LiDAR fusion	KITTI-style label files, JSON per frame
   Documents and PDFs	OCR correction, key fields, tables, layout	JSON with page coordinates
   Medical images	Organ and lesion masks in DICOM or NIfTI, reviewed by clinicians	DICOM-SEG, NIfTI masks
   LLM data	RLHF preference rankings, written answers for fine-tuning, rubric evaluation, red-teaming	JSONL prompt and response records

   Multimodal projects (an image with a caption, a video with its audio) combine rows. Ask for one sample export in your exact schema, including class mapping and coordinate conventions, before you sign anything.

3. 3

   Write annotation guidelines with edge cases

   • Takes: 1 to 3 days
   • Prevents: annotators guessing

   Annotation guidelines are the rulebook annotators follow: each label's definition, examples that are in and out, and what to do when a case is unclear. Vague or shifting guidelines cause disagreement between annotators, and the model then learns that disagreement.

   • One definition per label, with two examples that fit and two that don't
   • Edge cases written out: partly hidden objects, objects cut off at the frame edge, tiny objects, overlapping text spans, two people talking at once
   • A "can't tell" option and a named person who answers questions
   • A version number and a change log, so every batch says which rules it used
   • The class list (ontology) frozen before the pilot; changes after that are new work

   Keep the guidelines with the dataset. The "datasheets for datasets" practice (Gebru et al., 2021) is a good template for recording how, by whom and under which rules the data was labeled.

4. 4

   Choose a provider

   Choose how the work is staffed

   • Takes: a few hours
   • Prevents: the wrong kind of provider

   The staffing model decides who does the labeling and who runs quality control. Most data labeling outsourcing falls into one of these models, and many teams combine two.

   Model	Fits when	Watch for
   Managed annotation service	Large or ongoing volume, standard tasks, you want one accountable contact and their QA	How QA is really done, subcontracting, lock-in to their tool
   Crowdsourcing marketplace	Simple, high-volume tasks where several labels per item are affordable	Uneven quality, you run QA yourself, answers written with AI
   Expert network	Medical, legal or low-resource language work, RLHF and evaluation that needs judgement	Small pools, slower ramp-up, higher effort per item
   Staff augmentation	You have a lead who trains and reviews a dedicated team	You own the process, the tools and the guidelines
   In-house team	Data can't leave, labels need your own experts, volume is small and steady	Hiring, tools and management load
   AI pre-labeling with human review	A model drafts labels and people correct them (human-in-the-loop); active learning picks the items worth labeling	Reviewers accepting wrong drafts; check each class
   Synthetic data	Rare events or strict privacy limits	Still needs real, human-labeled data to validate against

5. 5

   Check security and compliance before you share data

   • Takes: 1 to 2 weeks
   • Prevents: data leaving your control

   Security and confidentiality cover who can see your data, where it is processed and what happens to it afterwards. Settle this before any file leaves your systems, including pilot data.

   Control	Ask for	Why
   Security management	The ISO/IEC 27001 (ISO 27001) certificate and its scope	A certificate that leaves out the annotation sites says little
   Independent audit	A SOC 2 Type II report	It tests controls over a period, not on one day
   Personal data and PII (EU, UK)	A data processing agreement under GDPR Article 28, processing locations, transfer terms	GDPR requires a contract with every processor
   Health data (US)	A business associate agreement under HIPAA (45 CFR 164.504(e))	Needed before protected health information is shared
   Access	Work in a locked remote desktop, no downloads, no personal devices, encrypted transfer	Fewer copies, fewer leaks
   Subcontracting	A list of subcontractors and annotator locations	The team you vetted may not be the team labeling
   Exit	Return or deletion of all copies, confirmed in writing	NIST SP 800-88 describes how media should be wiped

   Remove what the task doesn't need before upload: names, faces, license plates, account numbers. Put an NDA and an IP clause in the contract. If your system counts as high-risk under the EU AI Act, Article 10 requires documented data governance and names annotation and labelling among the data-preparation steps to record. The NIST AI Risk Management Framework is a useful checklist for US teams.

6. 6

   Set quality metrics that fit the task

   • Takes: 1 day
   • Prevents: arguing about "accurate"

   A quality metric is the number you accept or reject a batch on. Inter-annotator agreement measures how often two annotators give the same label; accuracy against a gold set measures how often they match labels you trust. Pick metrics per task and write the threshold down before the pilot starts.

   Task	Metric	Reference point
   Classification, tagging	Cohen's kappa (two annotators) or Krippendorff's alpha (any number)	Alpha of 0.800 or more is reliable; 0.667 to 0.800 supports tentative conclusions only (Krippendorff, 2004)
   Boxes and masks	Intersection over union (IoU) against gold labels	The PASCAL VOC benchmark counts a match at an IoU above 0.5; set a stricter bar if box edges matter
   Entities and spans	Precision, recall and F1 against a gold set	Score exact and partial span matches separately
   Transcription	Word error rate (WER)	Agree first how numbers, filler words and crosstalk are written
   RLHF preference pairs	Agreement on clear pairs, consistency of rankings (Bradley-Terry fit)	Disagreement on hard pairs is expected; don't force it to zero
   Evaluation, red-teaming	Rubric scores with anchor examples, coverage of a harm list	Check reviewers on the same anchor items

   Hide gold items in the queue so annotators can't tell them apart, and read agreement per class and per annotator: one person's bias shows up as one annotator disagreeing on one class. Check your own gold set too: a review of 10 popular benchmark test sets found label errors in at least 3.3% of items on average, and at least 6% of the ImageNet validation set (Northcutt et al., 2021). Several labels per item help on noisy tasks: on one affect-rating task, an average of four non-expert labels per item matched the quality of one expert (Snow et al., 2008). When a provider quotes "99% accuracy", ask what was counted, by whom, and for the spread across batches and annotators.

7. 7

   Run a paid pilot on your hardest data

   • Takes: 1 to 3 weeks
   • Prevents: finding problems at full volume

   A pilot (proof of concept) is a small paid batch that tests a provider on your real data before any long contract. Send the same batch to two or three providers, include your hardest cases, and agree the success metrics and a no-obligation exit in writing. The targets that pass become the SLA and KPIs of the contract: turnaround, accuracy against gold, rework. Ask for the profile of the people doing the work: training, domain background, languages.

   Pilot scorecard	How to measure
   Agreement with your gold set	The metric from step 6, per class
   Rework rate	Share of items sent back and fixed
   Turnaround	Days from data received to accepted batch
   Questions asked	Good providers find holes in your guidelines
   Format check	The export loads into your training pipeline without fixes
   Communication	Named project manager, time-zone overlap, response time

8. 8

   Compare quotes on total cost, not the unit rate

   • Takes: 1 day
   • Prevents: a cheap rate with expensive rework

   Data annotation outsourcing costs come down to effort per item, times the rate for the skill it needs, plus review and rework. Published dataset papers show how far effort per item swings with the method:

   Cost driver	Why it moves the price	Published effort example
   Task complexity and method	Outlines take longer than boxes, boxes longer than tags	Drawing a box the usual way took 34.5 seconds; "extreme clicking" cut it to 7 seconds (Papadopoulos et al., 2017)
   Objects per item	Many quotes scale by object, not by image	Polygon outlines for COCO took over 22 worker hours per 1,000 objects (Lin et al., 2014)
   Pixel-level detail	Full-scene masks are the slowest image work	Fine labels for one Cityscapes street image took more than 1.5 hours with quality control (Cordts et al., 2016)
   Labels per item and review layers	Consensus and extra review multiply effort	Four non-expert labels matched one expert on an affect task (Snow et al., 2008)
   Modality	Video multiplies work by frames; LiDAR adds a third dimension	Ask for effort per frame or per sequence
   Domain expertise and language	Clinicians, lawyers and native speakers of rare languages are scarce	Ask who reviews, not only who labels
   Turnaround and security	Rush jobs need bigger teams; locked remote desktops and on-site work add overhead	Ask for the price at two deadlines

   Providers price per label or object, per item, per hour, per fixed project or per dedicated team each month. Ask every quote to itemize setup and onboarding, tool licences, format conversion, project management, review and who pays for re-annotation when a batch fails. A low unit rate with heavy rework costs more than a higher rate that passes first time.

   Get quotes priced on your own data

9. 9

   Run the work

   Calibrate, then scale in versioned batches

   • Takes: ongoing
   • Prevents: drift between batches

   Calibration is a review session where you and the provider go through disagreements and update the guidelines before volume grows. Early fixes are cheap; late fixes mean re-labeling.

   1. First batch: review the first 50 to 100 items together, log every question and answer it in the guide.
   2. After calibration: publish guideline version 2 and re-label a sample to confirm agreement went up.
   3. Ramp-up: grow in batches; accept each one on a sample, using a sampling plan such as ISO 2859-1 that sets sample size by batch size.
   4. Steady state: monitor a short weekly report or dashboard with agreement per class, rework, open questions and the guideline version used.

10. 10

    Accept each delivery and keep the labels current

    • Takes: per batch
    • Prevents: stale or unusable labels

    Acceptance means checking a delivery before it enters training. A batch is accepted when it loads, passes the agreed sample check and comes with its paperwork.

    • Test-import every batch into your training pipeline
    • A file manifest, a quality report by error type and a log of skipped items
    • A fixed window to reject a batch, and who pays for fixes
    • A re-label plan for label drift: new classes, a new camera, a new product range or new user language
    • A contract clause that assigns the labels and guidelines to you
    • Deletion of your data when the project ends

Data annotation outsourcing vs in-house labeling

In-house labeling means hiring and managing your own annotators; data annotation outsourcing means buying the work from a provider. Many teams do both: their own experts write the guidelines and the gold set, and an outside team labels the volume.

The usual reasons to outsource are cost efficiency (no salaries or idle time between projects), a faster start, capacity that scales and an ML team that keeps its core tasks; the usual worries are control, security and quality assurance (QA). Where the team sits matters too. Onshore teams share your time zone and law; nearshore and offshore teams can cost less but need time-zone overlap for questions, clear rules on where data may be processed, and annotators who know the language and culture of your data.

Risks of data annotation outsourcing and how to cover them

Questions to ask a data annotation provider

Send the same questions to every provider so the answers line up.

1. Who will label our data: employees or contractors, where, and how were they trained for this task?
2. Who reviews the labels, and how many review layers are in the price?
3. Which agreement metric will you report, per class and per annotator?
4. Which tool do you use, and can you work in ours?
5. Can you export in our schema, and will you send a sample export now?
6. What is the scope of your ISO/IEC 27001 certificate, and do you have a SOC 2 report?
7. Will you sign our DPA or BAA, and where will the data be processed?
8. Do you subcontract any part of the work?
9. What is your policy on annotators using AI tools, and how do you check it?
10. Who pays for re-annotation when a batch fails acceptance?
11. How fast can you double the team, and what happens to quality when you do?
12. Who is our project manager, and how many hours overlap with our time zone?

Who outsources data annotation, and for what

Sources

• Artstein, R. and Poesio, M. (2008). Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4), 555-596.
• Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46.
• Cordts, M. et al. (2016). The Cityscapes dataset for semantic urban scene understanding. CVPR 2016.
• Everingham, M. et al. (2010). The PASCAL Visual Object Classes (VOC) challenge. International Journal of Computer Vision, 88(2), 303-338.
• Gebru, T. et al. (2021). Datasheets for datasets. Communications of the ACM, 64(12), 86-92.
• Krippendorff, K. (2004). Content Analysis: An Introduction to Its Methodology, 2nd ed. Sage.
• Lin, T.-Y. et al. (2014). Microsoft COCO: Common objects in context. ECCV 2014.
• Northcutt, C., Athalye, A. and Mueller, J. (2021). Pervasive label errors in test sets destabilize machine learning benchmarks. NeurIPS Datasets and Benchmarks Track.
• Papadopoulos, D. P., Uijlings, J. R. R., Keller, F. and Ferrari, V. (2017). Extreme clicking for efficient object annotation. ICCV 2017.
• Settles, B. (2009). Active learning literature survey. Computer Sciences Technical Report 1648, University of Wisconsin-Madison.
• Snow, R., O'Connor, B., Jurafsky, D. and Ng, A. (2008). Cheap and fast, but is it good? Evaluating non-expert annotations for natural language tasks. EMNLP 2008.
• Veselovsky, V., Ribeiro, M. H. and West, R. (2023). Artificial artificial artificial intelligence: crowd workers widely use large language models for text production tasks. arXiv:2306.07899.
• Partnership on AI (2021). Responsible sourcing of data enrichment services.
• Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 10: data and data governance.
• Regulation (EU) 2016/679 (GDPR), Article 28: processor.
• US Department of Health and Human Services. HIPAA Privacy Rule, 45 CFR 164.504(e): business associate contracts.
• NIST (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1.
• NIST. Special Publication 800-88: Guidelines for media sanitization.
• ISO/IEC 27001:2022. Information security management systems. ISO/IEC 5259 series (2024). Data quality for analytics and machine learning.
• ISO 2859-1. Sampling procedures for inspection by attributes.
• AICPA. SOC 2: Trust Services Criteria.