close

FSElite Original: The State of Reviews in Flight Simming

ORIGINAL State Of Reviews Flight Sim

With the release of Sky Force 3D from REX, FSElite naturally requested a press copy for our reviewers. However what we got in reply was that not only would we not receive a press copy, but a confirmation that neither us nor any other outlet would be receiving press copies for the first 30 days of its release. At that point, a significant number of people would have already purchased the addon. Of course, this would be long after anyone cares about a review for the product at all. This is not something that bothers us, but it is something that we feel our readers should know about. It’s also something that the consumers of Flight Sim addons should know as well, as they are the ones most affected. This article is written as a general state of reviews in this industry, and why they so often come out long after the addon in question.


The team here at FSElite knows that people want reviews, and we do read community comments – though I don’t always personally reply, I have read them all. We know that our readers want reviews on an addon’s launch day. Sometimes we are able to achieve this, but most of the time we can’t. The reasons why will become clear in this article.

Once REX released Sky Force 3D we requested our press copy, and we were subsequently denied. Now in every other industry, whenever a publisher does not give out press copies, it’s usually a red flag.  There’s almost always a reason behind it, and it’s typically a small story in and of itself. However, this is not an uncommon practice amongst flight simulator developers. Given that Sky Force was hyped up to be a game changing addon, we decided to take this as an opportunity to explain to everyone what exactly the state of press copies and reviews in the flight simulation industry is, and why exactly it takes so long to get a review done. 

What we want to make perfectly clear from the start that FSElite never expects a developer or publish to send a free copy. We have never seen it as a ‘right’, but mutual support for both sides to help expose their brand to the community. Equally, we would review a product in the exact same way whether we supplied the product or if the developer / publisher was generous enough.

 

The Nature of Press Copies

It should be known exactly what a press copy is and what it is for. I believe that it makes the most sense to look at flight simulation’s older sibling – gaming. Gaming media outlets such as Game Informer, Kotaku and many others typically receive free copies of upcoming games. These copies are given to the various reviewers at these outlets. This allows the reviewer to do their job at no personal expense.

Typically, press copies are given out in the weeks leading up to release, during a period where the game is largely done, and most bugs are fixed. Giving the press early access to a upcoming game is a point of contention for many, however, it allows the reviewer to play the game, and write the review – all before the game actually comes out. Usually a review embargo is placed on the press. The embargo is simply a date before which the press is not supposed to publish their reviews, or talk about the game in detail – often just before or on the day of release.

When the reviews are published at this time, it allows people to read them and decide whether or not they wish to purchase the game that day. It also informs a given outlet’s subscribers of the game’s release, and those subscribers can hear thoughts on it from someone whom they generally trust. This is the big payoff with press copies. The outlet gets to write a review, and the game publisher gets access to the outlet’s audience. Everyone wins, and it’s even possible that generally negative reviews pay off.

That’s how it works in the gaming industry, and the process is very similar with movies. Critics often go to special movie showings called press screenings. These are special viewings of a movie open only to the press, and they take place once or twice about a month or so before the movie comes out. Sometimes they even have catering. The reason these take place is for the same reason that the press gets early access to games; it’s beneficial to the critic, the outlet, the publisher, and – in the case of movies – the theater too.

By far, with both games and movies, the biggest benefit is had by the customer. And since the writers get early access, they can adjust their schedule to give the product the best possible chance. It also allows them time to write a full article and refine it to be a better read. Just for your information, the articles and reviews we write often take weeks. Usually at least two, and most articles and reviews go through seven or eight proofreading or refining passes before you see them. It should also be stated that multiple outlets normally get press copies, or get invited to press events, allowing multiple views on a single product. This gives a wider range of opinions for consumers to read, and a larger audience that publishers can reach.

Obviously, I am talking in a general sense. The above is how things happen ideally, but reality is sometimes cruel.

 

Press Copies in Flight Simulation

In the world of flight simulation, the press copy situation is very different. It is exceedingly rare for publishers to give the media early access to press copies. Often, the media does not get their copy of a product until release day, or sometimes well after release day. In my recent review of the Carenado C172, I did not receive the press copy until 3 full weeks after the product released. When you throw in the amount of time needed to test and use the product, plus the time it takes to write, proofread, and let the editor have a go at it, it can be months after release day before the review comes out.

This is not beneficial to anyone except the publisher. The review does not get anywhere near as many views as it could for the outlet, because no one wants to read a review on a product that came out months ago (apart from to confirm their already decided upon opinion). It’s not beneficial to the reviewer, because it does not allow us to adjust our schedule to give the product a fair chance. There’s a huge rush to get a review out as soon as possible for reasons that will become clear soon. It’s not beneficial to the product, or to the people that make it, since we have to rush through it in a effort to get things out quickly. And most importantly, it’s not beneficial to you – our readers – or consumers in general. This is because people want a product as soon as it’s available. It’s really hard to make an informed purchase off of nothing more than pre-release hype and a store page.

When you consider the increasing prices of flight simulation addons, and the lack of refund policies in this industry, reviews from trusted critics become very important. The only party who benefits from the current arrangement is the publisher, and nobody else. They get to have a bunch of uninformed customers buying their potentially overpriced product with no chance of a refund.

 

Why do press copies at all?

The question has to be asked: why exactly does anybody do press copies at all? Obviously it costs them money to give copies of a product out for free, even if it’s only to a select few people. So why exactly does anybody do it? Well, the aforementioned facts are true, but there’s more to running a company than money alone. Customer satisfaction plays a big role too – nobody wants to waste their hard earned money! Because of this, people want to make informed purchases. One of the best ways to do this is through reviews. If a customer feels their money is wasted then they are not leaving satisfied, and a dissatisfied customer base can kill a company slowly.

In the US a whopping 20% of small businesses fail in the first year, and 40% fail within 3 years. This trend is similar across multiple industries, and is unaffected by economic status. Now we can talk about why until the cows come home, and in my opinion the reasons are often simpler than you might expect. But honestly I think one of the biggest killers of any business – at least early on – is a focus on profit instead of stability. It matters not how much money your company makes if you’re standing in the unemployment line next year. A company that builds good relations with its customers will almost always last longer than one which doesn’t – a wise man builds his house on a rock, after all.

Speaking of wise men and building houses, a perfect example of how good businesses value community outreach happened recently. Contractor and YouTuber “Essential Craftsman” has a long project where he records each step involved in building a house. A speculation house actually, meaning he is not building it for himself but rather to sell it. A recent video in the series details the process of ordering the concrete needed to build a retaining wall, thus increasing the usable area of the lot. In the video he talks extensively about the batch plant (concrete supplier) he was using, Umpqua Sand and Gravel. In what has to be one of the best displays of valuing community over profit I’ve seen, Umpqua Sand and Gravel provided the concrete and rocks needed for the wall at no cost.

He ordered five trucks worth of concrete. One truck holds about nine cubic yards of concrete, and the national average cost for one cubic yard is $98. Do the math, and that company gave him roughly $4500 worth of concrete for a promo in a YouTube video. This is on top of the other community outreach projects the company does. Watch the video from Essential Craftsman linked above to hear about those.  

I told you that story to demonstrate just how important community outreach is to any company. Considering that 20% fail in their first year, and that this statistic is repeated across most industries, it really illustrates just how important community outreach is. There are a lot of reasons that companies fail. Cash flow is one of the biggest killers, and is a rather complicated thing in and of itself. But – to quote an old adage that I was told on the first day of my first job- the customer makes your paycheck, but it takes more than one purchase to do it.

Now to a game developer, or flight simulation addon developer, that means listening to customers, making sure the product you make is worth the money you charge (by your customer’s determination, not your own), working with press outlets to get word of new products and features out, and doing your part to make sure the press outlets get honest reviews out when the customers want them. But most importantly – don’t bite the hand that feeds you. For clarity, don’t treat the ones who write your paycheck with disrespect, for if you do, you won’t have a paycheck.

 

The Review Process

It should come as no surprise to everyone that we at FSElite do not do this as a job. We are all essentially volunteers like most other flight simulation organisations. This means we must balance our life and work, along with doing things for FSElite. Now I like to think we balance things pretty well, and I don’t want to come across as complaining – we all do enjoy doing this. And we are grateful that others want to read and watch what we do. But since we have to balance everything, it means we can’t give as much time to reviews as we might like. Especially when we don’t get our press copies until after release day. Because there is a real demand to get a review done as soon as possible, getting the copy so late means we may not be able to see or test everything.

It also means that if any unforeseen circumstance arise, and the assigned reviewer is not able to complete the review, the task is delegated to someone else who now has even less time to finish the job. This happened to us with Worldwide Airports. We planned to make two reviews; one on video and the other written. Patrick did the video review but some unforeseen complications led to the written one being delayed. I took up the task of writing it instead. From the point I got the press copy to the time the review was published was about a week.

I was helped out a lot by the others on the team here at FSElite who helped out with a good number of the screenshots. The fact the Worldwide Airports was a simple addon helped out too. Had that been a complex aircraft, or a large scenery addon there’s no way it would have gotten done as quickly as it did.

It may be beneficial to go over the review process for clarity’s sake. I can’t speak for everyone, but this is how I do it. My most recent review was the Carenado C172. The type of addon you review changes the approach, of course; for example, for aircraft I believe that visuals, audio, system accuracy, flight model, and performance are the main categories that people care about. In this world of increasing prices and stagnating wages, combined with just how competitive the entertainment business has gotten over the second decade of the 2000s, cost is becoming a big concern as well.

Typically I start taking notes the minute that the download has finished. I note down every point of feedback I have, starting with the install process. Depending on the addon, and the areas of focus, I have to change what I look for. When reviewing an aircraft’s flight model, I have to do several test flights in it. For performance, I often apply the scientific method, testing the aircraft in various locations and weather settings, and then repeating the tests with a similar airplane. Once all the testing is done, I go through the notes sheet and decide what’s worth talking about, and what’s not. Generally speaking, I tend to lean towards discussing the sum of the parts.  Of course, some parts are more important than others.

Of course, everything gets weighted relative to its cost – one must consider the amount of money spent, the time spent in the process of installation and fixing any problems, and what else you could acquire for the same cost. This is kind of a double edged sword, as the higher the price, the more functionality is expected, and the bigger the impact any single issue could have.

By the way,there are a couple of videos linked in the last section that I recommend everyone watch. They talk about reviewer biases and are things that everyone should know – writers, publishers, and readers alike.

Now all this can take time. The more time we have to devote to the review, the better; of course some of the time devoted is on us, but when you throw in the fact that we don’t really know when a new product will drop, or when we will get the press copy, it becomes difficult to devote the amount of time to a product it deserves. This is not made any easier when a product launches with problems.

Having press copies does make things easier even if we don’t get them early. Thankfully most of the publishers do give us press copies, and some do give us early ones. We are grateful for this, but I just want our readers to know why it takes so long after release day for reviews to come out. Generally out of 12 publishers; two give us early copies, six give copies on release day, two give copies after release day, one gave a copy to only one outlet (citing that only a single review -in German – mattered), and one gave no press copies or contact at all.

 

But to Be Fair

It’s not required of any publisher to give out press copies, and it’s not a right of any outlet to receive early access. Press copies and early access to them in other industries is a mutually beneficial arrangement, to the point that when it’s revealed that a publisher is not giving out press copies it becomes a instant black mark against the product. This is because usually when a publisher does not give out press copies it’s because they know there’s something in the product that won’t survive scrutiny, and they are trying to hide it for as long as possible.

At most, the lack of a press copy should be nothing more than an inconvenience to an outlet. I want to reassure our readers that the fact that REX is not giving out press copies to anyone is no big deal to us. As you know, our Official Review for REX Sky Force 3D came out a few days ago, and as indicated on the review page, in order to review the product we bought it. Though it was against REX’s wishes for any outlet to review the product until 30 days after release, we did it anyway. I doubt they like us anymore, or especially myself, but I’m not here to make friends, and that doesn’t change anything.

I should also say that not all addon developers do this. Most give out press copies when we request them. And some do give us early access to them. As written above the vast majority of that sample size does give out press copies. I can happily say that sample is representative of the whole. On the whole, the development and publishing community respects our review output and values the feedback we give. This has helped to build solid relationships and trust very quickly.

 

To really tear into it

I want to examine the statement from REX we received when requesting the press copy. The full quote is: “We are not offering any press copies at this time as we like to give time to let the product get out to the public and work through any issues that arise first.” Honestly, I think this is a cowardly way to say they don’t want reviews until after they’ve taken everyone’s money. Why that is, I don’t know. In my opinion, REX has yet to make a bad addon; even their World Wide Airports HD addon that I gave an average score to, I would not consider bad. However, this addon looks to be attempting something not done before.

Sky Force 3D, at least according to their own marketing, seems to attempt to bypass the built in weather engine of ESP based sims (excluding Flight Sim World), along with using actual 3D clouds instead of 2D sprites. Something of this nature is a risky endeavor, and a risky purchase for a consumer. Not only does it add another program to run in the background, it creates more potential failure points. New features also include taking advantage of the 10 day cycle built into ESP based sims, which is something that no other texture addon including previous REX products has done.

Given the experimental nature of the product, it’s even more important that consumers make an informed purchase. If there was ever a flight simulator addon that needed early press copies, this was it. The statement “give time to let the product get out to the public” is an outright misunderstanding of what reviews are for. They exist to help the public in their decision to buy a product. The press is not some exclusive club that get late access to an even more refined product. In other industries, particularly gaming, the early access comes with the acceptance that the press copy may have missing features or bugs. Not to the point of ridiculous, as by the time press copies go out, most features should be working, and most bugs fixed.

Developers: the public is not there to be your unpaid beta testing team, and reviewers are not here to stroke your egos. Developers, if you think a product is good enough for the public, then it certainly is good enough for the press too. If you think it’s complete enough to sell to people, then it’s complete enough to be criticised!

The other part of that statement  “work through any issues that arise first.” It seems that lately developers don’t know what Version 1.0 means. What I’m about to say applies more to Captain Sim whose recent 757 addon was such a trainwreck of a release that many people thought it was a beta. Version 1.0 is supposed to mean that all major and most minor bugs are fixed. Yes, the nature of software development means that there will still be bugs in a final release version. But not major ones, and certainly not game breaking ones. In all fairness REX SkyForce does work with no game breaking bugs on release day.

 

In fact I’d honestly say that this product demonstrates why press copies and early access to them are a good thing.

$44.95 is the MSRP of REX Sky Force 3D, by far the most expensive product from them. And it’s making some big claims and promises. This article was timed to release with our official review of the product by Calum. It became clear to both of us that Sky Force was not living up to the hype it had created. Read Calum’s review for more thorough analysis. My opinion on it is pretty simple, it’s only worth buying if you want it. The clouds are not really 3D, and in some cases don’t look as good as the clouds from REX 4. The product is so average I’d say that if you already have a texture addon like REX 4 then you can easily skip this one. Even if it would be your first texture addon, then REX 4 would be better as it comes with more textures to use. If you want Calum’s thoughts on it, then read his review linked above.

The community thoughts seem to agree. I’ve read a few forum and reddit posts where people were generally saying it’s okay at best.

 

All in all

I don’t think Sky Force 3D is a bad product, but it did not live up to the hype. It’s clear to me now that the reason REX did not want to give out press copies until 30 days after release is to ensure people brought into the hype before people caught on that it’s really not as good as their past products. The user reviews on this product so far have been relatively positive, but also many people seem to not be impressed either. Some are even reporting CTD’s and errors from the application. The statements made in this email, and the seemingly scheming nature of not allowing press access is something I believe that everyone in the community should be aware of. And this product, combined with how badly handled the CS757 release was, demonstrates exactly why. It should also be noted that when REX gave us a press copy of World Wide Airports HD, there was no 30 day silent period.

Obviously, I am biased in this. As a member of the press I do benefit from a free press copy, and benefit from one that provides early access. FSElite would benefit from them as well, as would other outlets when they have them. But the nature of the beast is that no review can ever be unbiased, or completely objective. We do our best to be fair but these are things that both publishers and consumers should know. Our jobs are made much easier by press copies, particularly by the fact we don’t have to spend our own money to do our job – but also by allowing us to work at a better pace. The result is a better review. It’s better for the consumer, and more fair to the product itself.

It’s not really frustrating that REX has decided to do this. It’s more disappointing really. Especially when you consider that REX makes good addons. Even their lowest scoring product is good. So one must wonder: is the potential financial gain of not giving out press copies worth it? Considering that most of the industry doesn’t provide press copies until release day anyway, from REX’s perspective they really lose nothing. The real loss is felt by consumers, and to a lesser extent, media outlets.  Even though we purchased a copy for review, we were not alone. In the weeks it takes us to write the review, even more people buy it. If the product ends up not being worth the asking price – and again REX and most other FS outlets do not offer refunds – then that creates a lot of angry customers. And remember, the customer makes your paycheck.

 

 
Tags : Flight SimReviews
Timothy Thomas

The author Timothy Thomas

You heard about the legend of Jesse James, Or John Henry just to mention some names. Well there’s a plane flying fella in the south today. A man name Tim from Atlanta GA. Except I’m not from Atlanta. But I do live deep in the swamps of Georgia. The high temps melted my brain, and all I have to keep company are the gators. There’s a lot of them. | View My Specs