pRotectionism: The way to compete in 2024

Have you ever...

...spent hours, maybe even days, tackling a tricky coding problem at work, only to discover a better, cleaner, and faster open-source alternative? UGH! There’s got to be a better way!

You're in luck!

Inspired by the populist policies of recent US administrations and a global resurgence in protectionist trade policies, we present to you the pRotectionism package for R. This package aims to level the playing field by extending the runtime of more efficient code.

Key Features

• taRiff(): Want to delay the execution of an efficient function? Simply call taRiff() on the function along with the desired delay time in seconds. Voila! You’re back in the game, competing against more efficient code on a more even playing field.
• autaRky(): If you are Jones-ing for that extra edge, autaRky() is for you. This function removes all imported libraries, providing you with a self-sufficient coding environment that just might help you keep your job.

Get Started

Download the package from GitHub: pRotectionism

Contribute

Interested in adding functionality? You’re more than welcome to contribute! Just be aware that any new functionality might incur an additional taRiff() at my discretion.

From Blood to Oil: A techies perspective on how the USAAF's pragmatism destroyed the Luftwaffe, dropped the A, and yet failed on it's Grandest Ambitions

  “I’ve given you the solution. It’s up to you to work out the details.” 

August 1943 marked a pivotal moment in the USAAF's air campaign against Nazi Germany. Over Belgium, just shy of German borders, American fighter planes reached their maximum range and were forced to turn back. This left the bombers to continue their treacherous journey unescorted — a dangerous situation as bombers, historically, had been susceptible to enemy interceptors. But the USAAF had a tactic in mind, one that didn't depend on fighter escorts.

Rather than relying on fighters, these bombers adopted a close 'box' formation. This allowed each plane's machine guns to cover its neighbors, creating a tight web of interlocking defensive fire. With over 300 bombers flying together, this formidable defensive approach aimed to fend off enemy air attacks and reach crucial targets. Their mission was twofold: cripple the Nazi economy and decimate the German Air Force (Luftwaffe). Central to this goal were the ball bearing plants and airfield manufacturing hubs in Schweinfurt and Regensburg.

The concept of using air power to devastate an enemy wasn't exclusive to the USAAF. World War I, marked by grueling attrition warfare, had armies searching for alternatives. With the birth of aviation, theorists across nations believed they had found their answer.

Imagine attrition warfare like two heavily-supported armies battling at the frontlines.

By targeting the enemy's economy and supply chains, it was theorized that one could debilitate an entire army. Some even suggested going a step further — attack the civilian population to break their will to fight. This doctrine, known as strategic bombing, was embraced not only by the US but also by Germany and the UK during WWII.

Strategic bombing aimed to render front-line warfare redundant by directly attacking industry or civilian populations.

What set the USAAF's approach apart were two distinctive features. Firstly, they persisted with daylight raids on specific targets. Both the UK and Germany had initially attempted similar daytime strikes, notably during the early stages of the Battle of Britain, but abandoned them due to substantial fighter-induced losses. Secondly, the US Air Forces, unique among WWII's strategic air campaign entities, remained under the Army's jurisdiction. This probably meant they had more to prove, fueling their determination.

The Schweinfurt was costly. Sixty bombers, each carrying ten crew members, didn't return. Despite the damage inflicted on the target, German reserves ensured production wasn't halted. Another attempt on Schweinfurt was made in October, once again without escorts, relying on the 'box' formation. Sadly, the outcome was much the same: heavy bomber losses amounting to 77 lost, nearly 20% of the attacking force.

Throughout this post, I’ll delve deeper into the strategies employed in the air war over Europe from 1943 to 1945, using aerial claims data from both the USAAF and Luftwaffe. This analysis will highlight how the USAAF weakened the Luftwaffe's relative and then absolute fighting capabilities.

Furthermore, a broader narrative emerges: the challenges institutions face when adapting to novel technologies. As a data scientist with experience in AI and machine learning, I've observed parallels in how teams within organizations navigate new technological terrains. The overarching lesson is the importance of adaptability and results-oriented strategies, as exemplified by the USAAF's pivot in 1944. (This thesis taken from To Command the Sky).

Aerial Claims Data

Luftwaffe Claims can be found here. USAAF claims can be found here.

Code for this post can be found here and here.

The foundation of this analysis is anchored in the aerial claims data from both the USAAF and the Luftwaffe during WWII. These claims, often referred to as 'victories' or 'kills,' are considered 'confirmed' based on the respective criteria set by each air force. Both datasets mention the pilot lodging the claim and the associated date.

However, recording methods between the two reveal discrepancies. The Luftwaffe's claims list seems incomplete, especially with noticeable gaps in the Eastern Front post-1945. Such gaps may be a result of lost records during the war's chaos. Furthermore, the Luftwaffe's standards for confirming claims appeared to be more lenient than those of the USAAF, potentially becoming even more relaxed as the war dragged on.

It's worth noting that aerial claims have a notorious history of exaggeration, sometimes by multi-fold, due to issues like double-counting or in-battle confusion. Despite this, I believe they offer a useful metric for comparing relative changes, shedding light on the evolving combat capacity of the opposing forces. My specific focus will be on the number of pilots logging claims within specific time frames.

The chart above provides a breakdown of the Luftwaffe's claims throughout WWII, with color-coding based on the nationality of the targeted aircraft. This data presents a comprehensive chronicle of the air war. After significant engagements like the fall of France and the Battle of Britain, the Luftwaffe's primary theater of operations transitioned to the Eastern Front in June 1941. This remained the focus until around the summer of 1943, after which the Luftwaffe began reallocating resources to counter the USAAF assaults over Western Europe.

A noteworthy observation is that the peak monthly claims occurred in the summer of 1943. Yet, a bulk of these were against the Soviet forces. Soviet planes were, on average, easier targets compared to those of the Western Allies. Hence, a decline in raw numbers doesn't directly equate to diminished combat proficiency. The Luftwaffe sustained formidable fighting prowess up until mid-1944, post which there was a precipitous decline in claims.

The questions this poses are intriguing: How did the Germans sustain their aerial efficacy for so long? And what transpired post mid-1944 that saw them receding from the skies?

A shift in the mental model

The setbacks during the Schweinfurt raids highlighted the USAAF's need for escorts. Several technological innovations were fast-tracked, including the introduction of drop tanks to extend fighter range and the P-51 Mustang, a longer-range escort that outperformed German propeller planes during the conflict.

However, tactical changes were equally vital. Initially, fighters were tasked with the primary duty of ensuring bombers returned safely. In effect, this meant they had to stay close to bombers, defending them. Strict regulations, like the rule preventing fighters from descending below 18,000 ft, gave German pilots an escape route: simply dive below that altitude. But by granting the fighters more flexibility, the USAAF could take the battle to the Germans.

These shifts resulted in a significant uptick in USAAF aerial claims, evident in the graph below.

The claims are categorized into the European (ETO) and Mediterranean (MTO) theaters of operations and color-coded by aircraft type. The bulk of these claims targeted German aircraft, although planes from Italy and other Axis powers are also included.

The 'first duty change' saw the USAAF claims in the ETO surge. Interestingly, this increase wasn't just due to the introduction of the P-51, indicating that the tactical shift alone provided a significant advantage.

While the USAAF did lose more planes in the first half of 1944, the proportion of losses relative to their total numbers declined sharply, attributable to the new tactics and the sheer volume of USAAF aircraft.

A pivotal realization was that high-altitude precision bombing wasn't as effective as hoped. Many bombs missed their targets. In response, the USAAF devised a new strategy: using bombers as bait to draw out and then decimate enemy fighters in the air.

(Schematic of new strategy was focusing on destroying enemy fighters in the air and using bombers as ‘bait’).

To compare the relative combat capabilities of the air forces, I've analyzed the number of pilots with claims over the last 30 days - a timeframe that offers a balance between reducing noise and highlighting short-term trends.

For the USAAF, I considered pilots making claims within the ETO/MTO, primarily against German aircraft, especially post-mid 1943 when Italy exited the war. For the Luftwaffe, I divided the data between those who downed a USAAF plane and their overall claims, given their multifront engagements.

The above plot indicates that, by early 1944, the USAAF had roughly matched the number of Luftwaffe pilots making claims against them. Considering the USAAF's more stringent claim standards, this could suggest an even more significant relative strength.

A subsequent plot detailing the first and last claim dates of Luftwaffe pilots reveals a stark contrast to the USAAF approach. While USAAF pilots were rotated out to train newcomers, German pilots mostly flew until they were incapacitated or killed. Thus, a pilot's last claim can potentially indicate their exit from the war.

A conspicuous surge in attrition is visible in early 1944, marked by pilots making both their first and last claims. This suggests that inexperienced pilots were hastily deployed.

A deeper analysis suggests that most of the pilot attrition during this period involved recent claim-makers. While German pilots in WWII were famed for their high claim counts (the top 100 fighter pilots are all WWII Germans), how did their elite fare?

When sorting pilots into categories based on their claims - 1-3, 3-5, 5-10, and over 10 - we observe that the USAAF had more pilots in the 0-3 and a similar number in the 3-5 category in early 1944. By the end of 1945, the USAAF matched the Luftwaffe in the 5-10 claims category but trailed in the over 10 category. Notably, the count of Luftwaffe pilots with over 10 claims surpassed other categories, save for 0-3. This discrepancy might arise from data preservation biases, where records of high-scoring aces were more likely preserved.

The data suggests that while the USAAF didn't boast as many top-tier aces as the Luftwaffe, the majority of claims were made by pilots with fewer than five claims. Elite pilots, while significant, weren't the deciding factor. Instead, the key was having a large contingent of competent pilots rather than a handful of elites.

The USAAF's strategy during this time was to wear down the Luftwaffe's vast pool of average pilots, thereby curbing their overall combat capability, if not outright defeating them.

Oil Campaign

Oil remained a pivotal resource throughout the war. Germany, lacking abundant domestic oil reserves, primarily imported its oil, notably from Romania, and also sought to seize it from Russia. In response to these challenges, Germany devised methods to convert coal into oil. This process, though costly, provided a crucial buffer against oil shortages. However, this lifeline became vulnerable when the USAAF began targeting these coal-to-oil facilities.

Starting in May 1944, the USAAF initiated a relentless campaign against German oil infrastructure, with significant impacts on German oil stocks and production manifesting within just a few months.

The above plot, sourced from the US Strategic Bombing Surveys, displays the interplay between consumption, production, stocks, and the USAAF's sustained assaults on oil facilities. There was a pronounced decline in production beginning in May.

Oil production facilities presented an optimal target for Allied bombers. These plants were large, making them easier to target from the air. Their intricate machinery was susceptible to damage and challenging to repair quickly. More crucially, oil was indispensable to the German war effort, and any disruption in its supply would significantly cripple their operational capabilities.

The strategy shifted to resemble a domino effect: by crippling primary (upstream) processes, subsequent (downstream) processes would inevitably collapse. This approach yielded stark results, evident in the noticeable dip in both the number of Luftwaffe pilots with recent claims and overall claims during this period.

Conclusions

This analysis has delved into three central strategic models the USAAF employed over Europe during WW2:

• - The belief that bombers could penetrate enemy defenses without escorts, primarily by targeting and decimating German factories.

• - The strategy of using escorts to directly combat and suppress the Luftwaffe in aerial battles.

• -The oil campaign, which identified and exploited a significant vulnerability in the German war machine.

Individually, these strategies might not have been sufficient to bring about Germany's defeat. In hindsight, certain foundational assumptions underlying these tactics were flawed. Yet, that's not the ultimate takeaway. What stands out is the USAAF's adaptability and pragmatism. Even in the face of overarching strategic challenges, they maximized their available technology and resources. These significant contributions and the evolving role of airpower in warfare paved the way for the USAAF to transition into an independent military branch. By 1947, they had dropped the "Army" designation, leading to the formation of the United States Air Force (USAF).

Appendix Plot
Targets and Losses of 8th Air Force 1943-1944

Are claims Accurate?

Looking at monthly figures of B-17 losses for the eighth Air Force between 6/43 - 7/44 (for which I have data for) I plot against claims of B-17s in Luftwaffe database. I use B17s since the majority of these were specifically used by the 8th Air Force in Europe and I have data on the 8th Air Force losses. Below we can see they track each other reasonably well. It should not align perfectly as the losses data are for all causes (accident, weather, flak, etc) not just fighters.

Interestingly regression analysis shows claims are ~half of losses so Luftwaffe is roughly double counting.

Monthly Movement and Losses of the Luftwaffe 1939-1945 Infographic

I created this infographic where you can see the monthly locations, movements, and for the middle of the war losses of the German Air Force in WWII. I cut the map into 2 latitude by 2 longitude squares and counted the number of aircraft within that square. I then color coded it by different type of aircraft. In the lower left of each square are the number of fighters (these include single engine, twine engine, and night fighters) and in the upper right of each square there are the number of bombers (either ground attack or general bombers).

Before February 1942 I estimate the sizes of groups as the data for number of planes are not available before than. Each vertical line heigh shows the number of planes maxing out at 100 before flowing over into subsequent lines. There can be 10 lines each representing a max of 1000. Starting February 1942 through December 1944 we not only have accurate data on the monthly number of planes at each location but we also see the monthly losses during that time period as well.

In order to visualize the movements and losses I based the graphic as if the data came from a turned-based strategy game. Here I visualize two phases; a movement phase followed by a ‘battle phase’ which consists of two additions to the original visualization showed. 1) the hot spots of the war and 2) the planes destroyed fall off into place in this time series plot at the bottom.

The full infographic can be found on YouTube (I cannot load the entire GIF on blogger). All data collection, formatting, and plotting was done in R. 

Luftwaffe Data came from ww2.dk

and Map data came from here

Code can be found here: https://github.com/samcarlos/luftwaffe_locations

For further information on this graphic please see this presentation

 

Visualizing the Invasion of the Soviet Union with Luftwaffe Locations

On June 22 1941 (81 years ago) Nazi Germany invaded the Soviet Union with the largest military force assembled in history. Behind the ground troops the Luftwaffe (German Air Force) quickly setup forward bases to support the campaign. Using data from http://ww2.dk I put together an infographic visualizing the monthly movements of the Luftwaffe during this time (map borders are current not the ones that existed in 1941). 

One can see planes flying into Poland from the west (Denmark, Norway, France, Belgium, and The Netherlands were conquered the previous year) and Greece (conquered the previous month) right before the campaign. As the Germans advanced along the Northern, Central, and Southern fronts one can also see the indecisive plan of attack. First they locate more planes to the North to attack Leningrad and then shift focus to the attack on Moscow. 

Stopped outside of Moscow in the winter of 41/42 the invasion did not succeed in taking over the USSR. Instead the conflict lasted another three and a half years and ended in May 1945 when the USSR took over Berlin - about 1000 miles from Moscow.

Code can be found here https://github.com/samcarlos/luftwaffe_locations

If you like this plot and want to see the first two years of the war through plane movements check out this post I wrote up here. 

Did the P-51 Mustang Defeat the Luftwaffe?

14 October 1943 -  291 US Army Air Force (USAAF) heavy bombers took off from England to attack a Ball Bearing factory in the heart of Germany. The goal was to destroy the Nazi war machine’s fighting ability and means of war production. As the bombers flew deeper into enemy territory their supporting escorts had to return to England due to the fighters’ shorter range. The unescorted bombers were vulnerable to enemy fighters and paid a heavy price. 60 did not return leading to a loss rate of over 20% thus cementing October 14, 1943 as ‘Black Thursday’. The ten crewmen on each of those planes had to survive 25 (later pushed to 30) missions to return home. They could not sustain losses like this indefinitely.

But they wouldn't have to for long. Within a year similar missions could be completed with far fewer casualties - often well below 5%. 

The above plot shows raids for the Eighth Air Force during two time periods; on the left the second half of 1943 and the right July - Mid August 1944. Each point represents a raid’s distance and percentage of planes that did not return back to base. This does not count bombers scrapped after landing due to heavy damage. Nor does it account for wounded, killed, or POWs in a mission (even if a plane came back not all of the crew did). 

In the chart on the left one can see a fairly strong if imperfect relationship between distance (I chose London as a starting point but airfields were scattered throughout southern England) and bomber losses. The second chart on the right shows this relationship is far less prevalent. The average loss rate of planes during the second half of 1943 was around 3.9% compared with 1% loss rate in the summer of 1944. Using these as proxies for survival probabilities and 30 as the number of missions required before a crew member finished a tour of duty the probability a member came home went from 30% to 74%. How did the loss ratio decrease by so much during this period?

The consensus story of those two plots goes something like this: after Black Thursday USAAF operations suspended long range attacks into Germany until a fighter was capable of escorting the bombers. Enter the P-51 Mustang. 

A P-51 with drop tanks. Notice the Swastikas underneath the canopy - the only acceptable place to display a swastika - each indicating an aerial victory. Having met the criteria of five aerial victories this pilot has earned the moniker of “ace”. 

An American aircraft (with an English designed engine) it was capable of flying to Berlin and back. Not only did it have the necessary range but also outclassed the German interceptors of the period. The P-51 was therefore able to ‘break the back’ of the Luftwaffe and gain air supremacy over europe. This allowed USAAF bombers to go into Germany and beyond without heavy losses.

But the claims of the P-51’s importance often goes farther than that. With air superiority the Allies launched the D-Day landings of Normandy unencumbered. And with Allies now fighting Germany in the west a Nazi Defeat was inevitable. A causal line of reasoning is therefore established between the P-51 and the ultimate destruction of Nazi Germany. Perhaps this is why - somewhat uniquely for a fighter plane - it’s achieved something of a totem status. 

But did the P-51 Mustang “break the back” of the Luftwaffe? This post will go over the consensus history of operations between the summers of 1943 and 1944. It will go over Mission Level data. An index is created of German Pilot Strength over time. 

This follows the history laid out by McFarland and Newton. But it seems most histories are similar to this and conclude that the Luftwaffe was defeated prior to D-Day. 

Strategic Bombing and Pointblank

Air power was still relatively new at the beginning of WW2. It wasn’t clear at the outset how best to use it in a military conflict. Broadly speaking there were two ways of thinking how the airplane could affect the course of the war. 

This first and most obvious use of airpower was as a tactical force. A precision armaments system that can provide ground support in real time to assist tank and infantry forces. The Germans seized on this use of airpower as a form of  ‘mobile artillery’ with devastating effectiveness. 

In contrast the English and American Armies focused more on Strategic Bombing. Instead of focusing and directly influencing a battle the bombers would fly deep into enemy territory destroying the means of waging war. The target wasn’t tanks and enemy defenses but enemy factories, oil refineries, transportation networks, and even civilian populations. However, an enemy has never been subdued by strategic bombing before. The USAAF got their chance to see whether they could destroy the enemy from the air with Operation Pointblank. 

To prepare for the D-Day Landings in summer of 1944 the Allied command issued the Pointblank Directive in June 1943. The objective was to destroy German Morale and ‘capacity for armed resistance’. It was to do this by imposing ‘heavy losses on Germany day fighter force’ and draw German fighters ‘away from Russian and Mediterranean theatres of war’. Once completing the objective the heavy bomber forces would be put to use in a supportive role of the D-Day in the Spring of 1944. 

For a brief overview of tactics in the European Air War see here. 

Attrition Warfare

Between the summer and through the fall of 1943 US bombers attacked German factories with high losses. After regrouping forces in late 1943 the allied offensive continued in early 1944. The new P-51 Mustangs were thrown into combat as soon as they could arrive during this period thus providing some cover to bombers at long distances. 

Despite the imminent need to destroy the enemy before the summer of 1944 they were often grounded due to poor weather. A break came through at the end of February with a forecasted week of clear weather. This was to be the start of Operation Argument (also known as “Big Week”) targeted specifically against the Luftwaffe in order to gain air supremacy. It’s objective was to minimise the ‘Production-wastage’ differential by attacking both the factories on the ground and Luftwaffe in the air. 

However, it became clear that the bombing offensive actually wasn't destroying industry at a rate where Germans could be kept up. Indeed during this period of increasing attacks German industry was put on a ‘war footing’ and increased fighter plane production despite the increase in bombing. In 1943 Germany Produced 25,500 planes. That number increased to 40,500 in 1944. 

But a realization occurred to US high command - even if the Germans could replace their fighters they could not replace their experienced pilots as easily. Instead of minimizing German ‘production’ they’d focus on maximizing the ‘wastage’ of german pilots in combat. They just needed to lure the enemy up to attack the bombers with the knowledge enough would be shot down by P-51s.

This was the logic behind attacking Berlin in early March 1944. The bombers would be used more as bait for the Luftwaffe than the bombing of industry. The USAAF did so with enormous costs. The largest one day loss of the eight air forces occurred on the first Berlin Raid of march 6 1944 with 69 heavy bombers lost and known as “Black Monday”. Due to the large number of aircraft dispatched this amounted to a 10% loss rate. 

These were losses the US could sustain. Luftwaffe pilot losses weren’t.  After the Battle of Berlin and heavy fighting the Luftwaffe changed tactics. They would only attack en masse and other times not confront the invading bombers at all. But ultimately they ceded air control to the USAAF.  The German Luftwaffe was defeated before D-Day. Or so the consensus history goes...

To look into this I gathered some data on Eight Air force missions and did some analysis. 

Eight Air Force Data

The original data set came from 8thafhsoregon.com/8th/. There is an excel spreadsheet that has missions broken out between the start of operations and August of 1944. This data includes (among many things) target location, number of planes dispatched and lost for both bombers and escorts. I used this as a template and then edited it as necessary, often looking at the WWII chronology on which it was based to fix any errors (also in that link). I’ve tried to be as accurate as possible but this was a manual process so there are probably errors. I appended Latitude and Longitude to the dataset using ggmap functionality in R. 

Some problems with the dataset are:

• Some names of places have changed since the original. German names are now in modern day Poland for instance.

• The location of dispatched planes are often ambiguous. Often there are several targets for a set of planes. I’ve no way to be sure which particular location each plane went to so I just chose one location for the lot. 

• After July 1944 escort by plane type wasn't broken out so I am unable to show the relative distribution of fighter types per mission. 

Eight Air Force Analysis

As noted previously a key feature of bombing losses was the distance travelled to the target. The longer the distance the more likely there were

The above plot shows a scatterplot of Distance travelled by Date for a unique raid location between January 1943 and August 1944. It’s colored and numbers by percentage lossed (red is higher, blue is lower). I’ve also removed missions that exceeded 800 miles for readability as they were few in number. 

One can see that loss rates decrease in both the time and increase with distance. Heavy fighting before winter of 1943 for distances more than 400 miles were particularly costly for the USAAF. These raids included Hamburg and the first two Schweinfurt Raids. One can also see a decrease in activity at the end of 1943 as weather conditions were poor and USAAF regrouped after those costly raids. 

Starting in February and March the USAAF went deeper into enemy territory - a period that covered both Operation Argument and the Battle of Berlin. While these raids sustained high losses they are represented in the low teens of percentages because of the larger number of bombers attacking. 

As Operation Pointblank ended focus shifted to targets in France and therefore smaller distances. In April, May, and June they attacked transportation targets, airfields, and sometimes more direct tactical support. 

Overall one can clearly see that as time progressed it became less and less costly for USAAF forces to fly far into Germany as a percentage of aircraft dispatched. 

Below is a plot of statistics by month to better see the aggregate losses more clearly. 

The above are aggregate statistics by month for heavy bombers: total dispatched, total lost and the lost ratio. 

Heavy monthly loss ratio existed prior to 1944 with a maximum of over 7% in October of 1943 (Black Thursday was the culprit). In part this was due to the small number of planes dispatched. 

Starting in February the number of planes dispatched increased to a maximum in June 1944 and then slightly back down in July 1944. Total lost planes increased throughout this period as well reaching a high point in April of 1944. Even in May 1944 the USAAF lost 300 heavy bombers. In June it dropped to 200 heavy Bombers. Why was there such a drop in losses between May and June? 

Looking at the increase in P-51 usage there was a sizable  increase in escorts p-51 more than doubling from April (3151 to 6844) to May but only slightly from May to June. Moreover there doesn’t appear to be an obvious relationship between increased P-51 Escort usage and decrease in Heavy Bomber Losses (at least on the aggregate data - see below for further analysis). 

The history I’ve described above suggested that most of the fighting was done in February and March. After this period the Luftwaffe was supposedly defeated and air supremacy passed to the USAAF. Looking at the loss ratio statistics one can see they were much more acceptable range for USAAF forces - especially compared with 1943. In some sense the Luftwaffe may have been irrelevant at that point in that the USAAF could expect replacements at that point to replace any lost bombers. 

But the evidence suggests that the Luftwaffe was still inflicting heavy damage through May and didn’t cease after March as the original history suggested. What happened? I think we need to continue looking into data from the German perspective and its relative strength over time. 

Luftwaffe Data 

 

I couldn’t find similar mission-level data for the Luftwaffe during WW2. (Who knows, maybe some of that data was destroyed by the Eighth Air Force?) I did find there is a good amount of data on Luftwaffe Pilot aerial claims and planes they shot down. Pilots had strong incentives to report the planes they shot down and such claims are probably overstated. However it’s a dataset that should provide some indication of fighting strength and activity over time. 

This Dataset comprises two parts.The first are the aerial claims for a pilot that includes (among other information) date, aircraft destroyed, and fighter group. The second includes data on when a pilot was removed from the war either MIA, KIA, or POW. 

Ater joining these two datasets to get precise entrance (first claim in database) and exit dates there are 2,565 pilots with a total of 50,419 combined claimed victories. As the average pilot in the dataset as over 19 claims it clearly is biased towards the elite pilots and doesn’t include those pilots who had no claims. However, this could still be a useful dataset to show relative strength over time. 

As an EDA I first looked at first entrances and exits by front over time. If a person transferred between fronts it would not show up in these charts. Note that I removed two entrance points from the graph as they were so extreme; May 1940 on West had 273 and June 1941 had 251 entries of pilots in the dataset. These correspond to the Battle of Britain and Invasion of Russia so we can expect a lot of pilots had their first claims during those periods. 

Also Note that the West involves mediterainan, north african, and southern europe where the Eight Air Force did not generally operate. They also include night fighters targeting against English bombing raids that were somewhat distinct from the day fighters against the USAAF.

On the Eastern front one can see exits and entrance follow a seasonality pattern that corresponds to German offenses summer offenses: Invasion of Russia in 1941, road to Stalingrad in 1942, and Battle of Kursk in 1943. 

For both fronts the Germans had Entries higher than existed throughout most of 1943 and early 1944. Only after the summer of 1944 were losses consistently higher than entries  for both fronts.

On the Western form there was a lot of activity for the Battle of Britain - this was the Luftwaffe’s failed attempt at strategic bombing. Starting January 1944 there was a sustained increase in exits corresponding to the attrition warfare of the Eighth Air Force discussed above. While there was clear damage to the Luftwaffe during this time it reached a maximum loss of 66 pilots in July second only to losses in December 1944. If the Luftwaffe was defeated by the USAAF before D-Day then why were they losing so many pilots after that? 

Luftwaffe Defeated at Normandy?

As mentioned above the Luftwaffe was a tactical air force designed to support ground offenses. German command clearly intended to use it as such during the expected Allied invasion of France as they moved a large number of Fighters towards France after June 6, 1944. 

To better understand this dynamic I created an index of fighter strength by day for Reich Defense with the pilot entrance / exit and claimed data. For a particular date a pilot was included in the dataset he had claimed an aerial victory before that date and did not exit by that time. We can also see, from the most recent claim, which fighter group he was in. Using this we can tie a particular pilot to a location and therefore if he was actually under the Reich Defense.

Determining which Fighter Group and squadron were actually defending the Reich at a particular point in time was a manual process greatly helped by this website. It showed that most fighter groups defending Germany moved to France immediately after the allied invasion of Normany in June 1944. For example III Gruppe of JG 54 went from Illesheim, Germany to Villacoublay, France on June 7 1944. Moreover one can see (but not shown here) the claimed planes shot down after June 6 changed from US heavy bombers to short range english fighters (Spitfires and Typhoons) adding further evidence of the shift in priorities. 

Above is a plot of the number of Pilots defending the Reich compared to all other fronts combined between Summer 1943 to October 1944. I narrowed down the Dense of Reich to those that were likely to face the Eighth Air Force in Battle so did not include fighter groups in Italy or Nordic countries. One can see it remains relatively constant up through the spring of 1944 when it actually shows an increase in the relative number of pilots. By the early summer over half of the pilots in this sample were in Defense of the Reich. It then dropped dramatically when Day Fighter forces moved to France following the invasion of Normandy. 

But wait, an increase in pilots when there’s heavy attrition fighting in the first half of 1944? How is this possible? 

To check I created a stacked barchart by time. If the Pilots exited it counted a negative number and if they entered it counted as a positive number. One could either enter from another front or be a new pilot. They could exit to another front or exit the war entirely. (For an additional check see 2) below in the appendix. )

While there was an increase in pilots exiting throughout the first half of 1944 it appears that was offset by a combination of new pilots and reallocation of pilots from other fronts. There were few pilots exiting to other fronts until June of 1944. While it seems hard to imagine the fighting strength of the Luftwaffe increasing significantly during this time the evidence suggests that the USAAF did not destroy the Luftwaffe prior to D-Day.

Assessment of P-51

But that doesn’t mean the P-51 wasn’t instrumental in defending the bombers. I created a regression for the time period before June 5 1944 using data from the Eighth Air Force and combined it with Luftwaffe data. The response variable of interest is the number of bombers lost in combat for each unique raid. 

The data generally gave precise locations of where the bombers went; it did not give similar information of where Escorts went for all missions. For example, Mission Number 367 on May 24, 1944 USAAF sent 616 bombers to Berlin and 490 bombers to French Airfields. They were escorted by 144 P-38’s, 178 P-47’s and 280 P-51’s. While USAAF would probably have sent a majority of P-51’s to Berlin there’s no way to encode that. Instead I assume an weighted percentage of fighters going to each location proportional to the relative size of the bomber fleet. In this case I calculate a weighted_p51 score of 156 (616/(616+490)*280 ) escorting the Berlin Bombers.

Using the Luftwaffe Claims Data I am able to calculate the sum of all claimed kills for pilots defending the Reich on any particular day. This is something akin to the liberty ship ‘learning by doing’ literature. I’ve found this to be the strongest predictor for any single Luftwaffe pilot data. 

I include date as a continuous feature and also include dummy variables if there were no particular escorts. 

Below are the results. 

One can see that distance is a very important feature but that time isn’t statistically significant. This is promising as it shows there isn’t a trend that cannot be explained with features given. Also note that the total_kills variable - a proxy for strength of luftwaffe - is positively correlated and has a p-value of .06. While it’s not a strong variable in the regression it adds some credibility as a feature in general. 

Of particular interest is that the P-51 is negatively correlated with a p-value of .02. This indicates that the more P-51 escorts there were the fewer the bombers lost. Interestingly we also see a somewhat positive correlation with regard to P-47’s. This could be due to the fact that Pilot’s transitioned from P-47’s to P-51’s and the regression is picking up on that trend.

Conclusions

While evidence presented here is contrary to the general story of the Air War over Europe I believe it displays a more nuanced and realistic version of events. The Eighth Air Force did not defeat the Luftwaffe but it clearly made an impact and drew resources away from other fronts - a key objective of the Pointblank Directive. Moreover it made sure the Luftwaffe was tied down in Germany and not defending France while D-Day occured. It also seems clear that the P-51 was instrumental as it allowed the Eighth Air Force to launch attacks deep into Germany while maintaining acceptable losses. 

But it also seems clear that the Luftwaffe threw a large number of fighters from defending the Reich towards defending the western front and lost many pilots as a result. It wasn’t so much that the Eighth Air force defeated the Luftwaffe so D-Day could happen. Instead it seems like D-Day drew Luftwaffe resources away from Reich Defense so that the Eight Air Force could fly more freely over Germany after June 1944. 

Appendix 

1. 1)

Thought I’d just add this chart that shows the types of targets the Eighth Air Force attacked during that period. Initially U-Boats and naval sites were a large target but after they became less and less of an issue they focused more on aircraft and industrial targets in Germany in early 1944. During that time they also attacked V-Weapon sites. I thought that was interesting since it wouldn’t be until the summer when they were operational. Towards the end of the sample period they started attacking oil and even some tactical support in France.

2) As an additional check to using this data I compared the monthly exit rates by a dataset at the end of To Command the Sky (curiously they didn’t make a similar plot in their book).  While the index I created using Tony Wood’s focuses on Pilots this dataset from TCTS looks at the number of planes shot down. They should be correlated and that is what we see below. However the scale of TCTS data is almost an order of magnitude higher suggesting that my index is missing a lot of data and is incomplete. I think it’s ok as a relative changes but probably not as levels 

Works Consulted

Big Week: BIggest Air Battle of World War II - James Holland 

To Command the Sky - Stephen L. McFarland, Wesley Phillips Newton 

Engineers of Victory -  Paul Kennedy 

DOOLITTLE, BLACK MONDAY, AND INNOVATION - J David Rogers

How the Mustang trampled the Luftwaffe - Robert W. Courter

The P-51 Mustang: the Most Important Aircraft in History?  - Marshall L. Michel 

The Battle of Britain, in 1940 and “Big Week,” in 1944: A comparative Perspective - Arnold D. Harvey  Github