Breaking a 15,000-row cap

The Soldier in Later Medieval England database at medievalsoldier.org holds service records for English soldiers from 1347 to 1459. Each row is one man in one document: his name, rank, captain, the campaign or garrison, the date, and the manuscript it was transcribed from. The search form exports results as a CSV, and my first export of the whole database came back at exactly 15,000 rows. That's a server-side cap. I got past it by pulling the database one year at a time, 121 windows from 1340 to 1460, none of which reaches the cap, and deduplicating on all 12 fields of each record. The result is 270,816 unique service records in a local SQLite database.

With every record in one table, two questions about the army have direct answers. The army of three archers to every man-at-arms is a pattern that appears after 1395: the archer to man-at-arms ratio is 2.50 in the 1350s, 0.89 in the 1370s, and 3.33 in the 1390s, and it stays between 2.77 and 3.02 for the five decades after. And recruiting was concentrated: 55 captains, 2.7 percent of the 2,022 named in the records, commanded 44 percent of all recorded men.

This covers the extraction, the composition of the army by decade, the concentration of command, and the five records with women's first names. It leaves out pay, because the export carries no wage fields. It also leaves out identity: a row is a service record, not a person, and nothing here links one man's rows together.

The Cap and the Year Windows

The export isn't a plain link. The search form posts its parameters with downloadds=1, the server writes a CSV to disk keyed to the PHP session ID, and a separate GET to /database/downloads/download_ds.csv returns the file. I found that last step by driving the form in a headless browser, which showed the "Click here to download your file" link the post produced. After that a scraper needs a session cookie, the post, and the fetch.

The cap applies to one result set, so the way around it is to ask for result sets that are each smaller than 15,000. The form takes yearmin and yearmax, and one year of recorded English military service is never that large. I pulled each year from 1340 to 1460 separately, with a one-second sleep between pulls and all three source subsets switched on (datasets=111: the National Archives records, the English garrisons in France from 1415 to 1450, and the letters of protection and attorney). The scraper flagged any file that came back at the cap so that year could be split further. None did. The largest window was 1415, the Agincourt year, at 12,603 lines.

The 121 files overlap with the first capped download, and they also contain exact repeats inside a single year. In 1372 a letter of attorney for Henry Grene, a knight in naval service under Edward III (TNA C76/55, m. 17), appears twice in that year's file, identical in every field. The same file has two other Henry Grene records that year. One is a letter of attorney dated 4 February for service in France, and it is also in the file twice. The other is a letter of protection under John of Gaunt. Name and year put all three documents in one row. I keyed on the whole record instead, all 12 fields. This is a reduction of the combine step:

import csv, globFIELDS = ["Firstname", "Surname", "Origin", "Rank", "Status", "Service",          "Captain", "Lieutenant", "Commander", "Service Date",          "Source Type", "Reference"]seen, rows = set(), []for path in sorted(glob.glob("chunks/year_*.csv")) + ["download_ds_capped_15000.csv"]:    with open(path, newline="", encoding="cp1252") as f:        for rec in csv.DictReader(f):            key = tuple(rec[k] for k in FIELDS)   # all 12 fields            if not any((v or "").strip() for v in key):                continue                          # the export's blank footer line            if key not in seen:                seen.add(key)                rows.append(rec)print(len(rows))

Rows that match on every field are one record. Rows that differ in any field, even only the membrane number in the reference, stay separate. That leaves 270,816 rows, and the dated ones run from 28 August 1347 to 18 July 1459. I checked a sample against the live search on the site and every field matched.

The loader writes them into one SQLite table with the 12 source fields, a year, month and day parsed from the YYYYMMDD date string, and ten indexes on names, captain, commander, rank, status, service, year, date and reference. The file is 102 MB. The parser only reads the eight-digit form, and 33,580 rows have no parsed year. Only 142 of those have no date at all. The other 33,438 start with a year but not in the eight-digit form: 25,359 are a bare year, the rest are ranges, seven-digit strings or years with a question mark, and 7,620 say only "1415". So the deduplicated count for 1415 in the dated table is 1,618, and the peak year among dated rows is 1431, at 10,401.

The table has no person_id. 26.7 percent of all records are named John, and the same man shows up as Jehan and John, Robin and Robert, Ferrariis and de Ferrars. The plan in my follow-up notes is a canonicalization layer first (strip particles like de and le, map Latin, French and English forms, resolve diminutives), then phonetic matching within the same year, captain and rank, then a person_id from the resulting clusters. Everything below counts rows or distinct name pairs.

The 3:1 Army Starts in the 1390s

Rank is free text with variants ("Archer", "Archer (mounted)", "archer?", "Man-at-Arms (foot)"), so the count matches on the prefix. 27,486 records have a blank rank and fall out of both columns. This is the query and its output:

SELECT (year/10)*10 AS decade,       SUM(rank LIKE 'archer%')      AS archers,       SUM(rank LIKE 'man-at-arms%') AS men_at_arms,       ROUND(1.0*SUM(rank LIKE 'archer%')/SUM(rank LIKE 'man-at-arms%'), 2) AS ratioFROM soldiersWHERE year BETWEEN 1350 AND 1449GROUP BY decade;

decade  archers  men_at_arms  ratio------  -------  -----------  -----1350    704      282          2.51360    2609     1757         1.481370    9180     10363        0.891380    9651     8505         1.131390    4013     1205         3.331400    5627     2007         2.81410    2134     770          2.771420    21319    7682         2.781430    58497    19861        2.951440    31513    10444        3.02

The 1350s start archer-heavy at 2.50, on a small base of 986 men. The 1360s fall to 1.48. In the 1370s men-at-arms outnumber archers, 10,363 to 9,180, and the 1380s are close to even at 1.13. Then the 1390s jump to 3.33, and from the 1400s through the 1440s the ratio never leaves the range 2.77 to 3.02. The 1390s figure rests mostly on one year. 1395 has 3,199 archers and 854 men-at-arms, while 1390 through 1394 have no archers at all and 93 men-at-arms between them.

The strongest objection is that the 1370s dip comes from the sources rather than the army. The decade has 6,111 letters of protection, and protections went mostly to gentry, the class that served as men-at-arms, so a decade heavy in protections will show more of them. I reran the query with every letter of protection and letter of attorney removed. The 1370s ratio moves from 0.89 to 1.06 and the 1380s from 1.13 to 1.39. The 1390s move the other way, from 3.33 to 3.74. With the protections out, the two decades before 1390 still have fewer than one and a half archers per man-at-arms, and the five decades after 1400 are still near three.

My notes treat the shift as real and leave its proximate cause open: Edward III's death in 1377, Richard II's reign, and Bolingbroke's seizure of the throne in 1399 are all on the list. The 1395 jump comes after the first and before the last. On these records the archer-dominant army belongs to the decades from 1395 on, most of them Lancastrian.

Fifty-Five Captains

After the placeholder values (*[no captain given], *[name lost], [Captain's name missing] and similar, 5,737 records under ten values) are filtered out, 2,022 distinct captains remain. Grouped by the number of men each commanded across the whole dataset:

The 53 captains in the last row commanded 116,844 of the 270,816 records, which is 43 percent, and the next two bring the top 55, 2.7 percent of all captains, to 118,791 records, 44 percent. Add the 59 in the row above and 112 captains, 5.5 percent of the total, account for 58 percent. English armies of this period were raised by captains who indentured with the crown and brought their own retinues, and in these records a few dozen of those retinues carry almost half the army. At the other end, 396 captains appear with exactly one man.

The single most-recorded English commander is John Talbot, earl of Shrewsbury, with 9,195 records across 22 years between 1407 and 1453. Second is John Beaufort, duke of Somerset, with 6,693 records packed into six years, 1439 to 1444. His records hold 5,630 distinct first-name and surname pairs, 1.19 records per name. 94 percent of those names appear in only one of his six years, and nobody appears in all six. Retention from one year to the next was 18 percent from 1439 to 1440, then 1, 0, 1 and 8 percent. His 1443 Maine expedition mustered 4,228 distinct men, and 56 of them, 1 percent, had served him the year before.

Talbot and Fastolf look different on the same measures. Talbot's records run 1.26 per name with 91 percent appearing in a single year, and one man serves him in nine different years. Fastolf's run 1.33 per name and 90 percent, with one man across eight years. Those are households with long-serving retainers. Beaufort's six years of records are six separate mobilizations, each raised for its campaign from men who had mostly never served him. Name collision means these figures understate the turnover. A John Smyth in 1439 and a John Smyth in 1443 may be two men, so the real carryover is thinner than 1 to 8 percent.

"Jehan", the French form of John, appears 311 times in the 1420s, 2,360 times in the 1430s, 677 times in the 1440s, and not at all in the 1450s. My notes read the 1430s spike as the Bedford regency in Paris, with French-speaking clerks writing English names in French form. The zero in the 1450s lines up with the loss of Normandy: the whole dataset drops from 1,039 records in 1449 to 48 in 1450.

The Five Women

My first search for women's first names returned five records. Three are letters of attorney in the National Archives C76 chancery series: Alice Stapelton in 1376, Eleanor Cheyne in 1378, and Maud Well in 1380. A letter of attorney let someone manage an estate while its holder was overseas on campaign, and these are filed with the protections because of the military context. They are women handling property for men at war. The fourth, Joan Pecart, is a foot man-at-arms in the Rouen castle garrison in 1444, from a French manuscript (BM Rouen, Ms. Martainville 198). Joan is most likely an English rendering of the French male name Jean, and Pecart looks like Picard.

The fifth is Mary Samond, a man-at-arms on Henry IV's invasion of Scotland, dated 24 June 1400, in a retinue list at TNA E101/42/16, m38. Her captain, John Haddley, has 145 archers and 67 men-at-arms that day, a normal-shaped retinue, and he appears as a captain nowhere else in the database. Samond appears nowhere else either, though Salmon and Samon do, all men. The database lists the membrane's 67 men-at-arms alphabetically by surname, and her row sits among the S names. The site's own first-name variant search doesn't map Mary to any male form. A woman serving in disguise would have enrolled under a man's name, so a female combatant listed as Mary contradicts itself, and my notes rank a misread or misexpanded abbreviation of a male name as the most likely reading. The database can't settle it. The membrane at Kew can, and reading it is on my follow-up list.

Limits

The transcription error rate is unknown. The records are a modern project's transcriptions of medieval clerks' work, and the plan to measure it is to spot-check five to ten of the largest manuscripts against the originals, starting with TNA E101/48/17 and its 1,623 men.

These are service records, not a roll of every soldier. Coverage varies by year and by source: muster rolls are about two thirds of all records, the 1370s lean on letters of protection, the early 1390s are nearly empty, and the 33,580 rows without a parsed date are absent from every per-year and per-decade count above.

The 12-field key can fail in both directions. Two distinct men with identical records in the same document merge into one row, and one man written two ways stays two rows. The person_id layer is the fix, and it isn't built yet.

The Shape Changes in 1395

The site was built so a historian could look up a name and see where that man served. Pulled whole into one table, the same records show men-at-arms nearly matching archers through the 1370s and 1380s. The three-to-one army first appears in 1395, eighteen years after Edward III's death, with 3,199 archers to 854 men-at-arms that year, and the ratio stays between 2.77 and 3.02 through the 1440s.