Developer Spotlight: Chad Goodman aka MasterChefChad of AnthraX Kernels

In this installment of ROM Developer Spotlight, we ask Chad Goodman of Anthrax-Kernels.us a little bit about himself and his development life. For those who are not familiar with Chad, he’s written many kernels for the HTC EVO line of phones, along with some that work on other devices.

For the past year or so, Chad has had a running history of being one of the most interesting developers to watch – both on the development front, and in clashes with other developers.

For our readers who don’t know who you are in the development community, how would you introduce yourself?

My Name is Chad. My programming experience started out on AS/400 systems running COBOL, RPG-IV, CICS. I never got into the Java side of things. My experience grew into C and C++ while working on closed-source kernels for IBM AS/400 systems. We did not have an open source community to “cherry pick” from, so we were forced to do real development instead of relying on the work of others.

I also have experience as a network engineer, a data center manager, and I hold various Microsoft and hardware certifications. Prior to my career in information technology, I was in retail management for 5+ years, as well as an auto mechanic (until a back injury).

Have you ever developed ROMs? If so, which ones?

In the past, I did basic theming – no actual development. I have been working on a project over the last month or so, and I released it the other day. I released the first (and only) fully Linaro optimized CM10-based ROM  with -O3 optimization for a HTC device. I searched high and low for any Linaro optimized AOSP or CM10 rom for any HTC device, and have not found any. Today, we have a Shooter (HTC EVO 3D) version.

Can you give a quick rundown of what Linaro and –O3 mean to the end user?

Linaro is a group of open source developers. They develop toolchains (compiler), as well as AOSP ROMs and kernels. They are known for having the most optimized and cleanest code, as well as the best performance. Most of my learning was from reading their commits and patches.

Compiling with Linaro will generate errors and warnings on most kernels; its a “strict” compiler, so to speak. There are two ways to approach this: either just fix the code causing the error (what most developers do, including my public kernel source), or optimize the code where the errors/warnings exist, as is done in the Linaro repo (this is what my close source kernels do – some of this took a long time to do, since linaro kernels are now at linux 3.6 vs linux 3.0 used in my kernels).

-O is an optimization level. There is -Os (optimize for size), -O1, O2, -O3 (most optimization), and, depending on your toolchains, there is -Ofast. -O3 is the highest level of optimization. -O3 is also known to cause issues if the source itself is not optimized, but on clean source, end users report a noticeable increase in performance.

How did you get started in kernel design?

I did it for a living for closed source hardware. While it was not Linux, it was similar. I decided I wanted to make a kernel for my phone, and add features no other kernel had – features such as audio enhancements, better audio streaming over Bluetooth, and just massive optimization, to increase battery life and performance without the need to increase clock speed. I did this, gave it to a few friends who pushed me to post it on XDA, and it just took off from there. I do want to thank ziggy471 for getting me started on the basics of Linux development, as well as showp1984 for introducing me to the Linaro toolchains. It took me a while to make a -O3 compliant kernel using Google’s NDK toolchains to work with Linaro toolchains, but the effort was well worth it.

How did you get involved in the EVO community?

I started in the EVO 3D community because it was the device I owned. My kernels were started as a personal project to run on my phone; I did not intend for it to be available to the masses.

The products you release currently require signing up for your website and becoming authorized before downloading. Would you mind explaining a bit about what lead to this?

This has to do with the requirements to be GPL compliant. The GPL states that kernels released within a private organization are not required to comply with GPL, unless the kernels are released to the “general public”.  My keeping my site a “private organization” allows me to, from a technical point, be 100% complaint with GPL’s policies and rules.

Some wonder why I am against releasing source, and I want to clarify some things on this. Prior to me posting kernels on XDA, I got the approval from XDA’s administrative staff to post without complete source, as some code was released to me under a NDA. Just like the kernels for the HTC EVO 4G LTE do not contain the WiFi drivers from HTC, my kernels omitted source for some audio drivers produced by Beats Audio (Monster Audio). An employee that represented these companies released the source to me under a NDA, so I made use of it (with XDA’s permission).

In my first kernels, I was mainly pulling updates form source code form other devices, and my focus was 100% on audio. Some GPU and performance changes I made came mainly from the Vigor (HTC Rezound). Other developers at this point knew my Bluetooth and audio drivers were superior to anything XDA had to offer, so some accused me of kanging their source. The source that was in question was GPL drivers; that was a line-for-line match to the Rezound drivers at htcdev.com at the time. It was later discovered that the false accusations against me was an attempt to force me to release full source to prove I did not kang. Due to the amount of drama this caused, I decided I would actually pull all my work from XDA and not share anything, mainly due to the community standing behind the other developers.

You’ve been one of the most polarizing developers in the EVO community. What’s it been like being seen as both the bad guy and the good guy to scores of people, depending on who managed to spin it?

Its actually quite irritating. When all this drama started, staff members on XDA decided to hide posts, screen shots, and other evidence that I was innocent of what I was accused of, yet posts accusing me of wrong doing were allowed to remain. I feel some of the staff at XDA is biased.

People who defended me got “infracted,” or even banned. While people talking trash against me got “warned,” the punishment was far harsher for those in favor of defending me.

I was even called a “c***,” “idiot,” and “scum” by an elite recognized developer who had no business even being in HTC EVO 3D section on XDA, and XDA’s staff thanked that post. That tells me no warning or infraction was issued. That post was reported and no one removed it. It took two days after the main XDA forum administrator was asked to address the issue.

Someone said “Chad had that first, how did Chad kang you?” and got infractions within minutes. Google’s cached pages were my proof of innocence, but these are only available for six months, so today, most are gone. People who openly admit they witnessed the drama are label “liars.”

While XDA probably matters some due to the large developer and test-bed audience, has being away from the place adversely affected your development efforts?

Being away from XDA has not really affected me. While most think I was banned from XDA, I want to make it clear that I asked to have my account, my work, and all my threads removed from XDA several hours before the general population thought I was “banned.”

As I stated earlier, I do this for me, not for the users. I do it to prove to myself that I can do things no other developer on XDA has been able to do (fixing things and making things work without source to work from). What I made, I use. Feedback from users just helps me improve things. While I created and invented the modification that allows you to control the sweep2wake modifications, it was a request from an end-user that made me create this modification.

You had a long and very public fight with another developer that evidently involved a third party launching an attack on your website with police involvement ensuing. Can you go into any detail on that?

One of the administrators of my site is a forensics investigator for a nationwide internet service provider in their network security department. I myself also have a few years experience in network security. Together, we were able to get the IP addresses of people who attempted DoS attacks on my sites.

Using our connections with various internet providers, as well as a lawyer I am personal friends with, we were able to translate this IP address to a account name and address, and for domestic accounts, we filed charges.

Some investigations are still on-going, so I can not go into further detail on this at this time, other than to say there have been several arrests, and some of the arrests turned in to “guilty” pleas in a court of law.

Not to go into detail, but the reason that at least one post was required before people could download is to get their “real” IP address. When a user registers – we track and log their IP address, but this IP address may be that of a proxy server.

We have a script that will pull both the “real” IP address as well as the IP address of the proxy server, if a user is connecting via a internet anonymous proxy server, or using some other method of hiding their “real” IP address.

What do you think is the greatest misconception about you?

Other developers have told the community that I am in fact breaking GPL, and that I have kanged their work. Due to how “popular” and “well known” these other developers are, the community, as well as some staff on XDA, labeled me as a kanger (one who uses code from other people, and claims it as their own).

This “label,” as well as all the false information being posted about me has caused issues in real life, mainly with getting a job as a software developer.

I wish people would do their own research on kanging and GPL policies and not just find me guilty, because “developer x” is a “big developer” or more well known, or a long time favorite of the community.

To this day, a team of develoepers is still pushing the community to label me a kanger, yet no one can prove it. There is proof that I did in fact have the kernel first, but some people decided to hide this from the general public.

I actually had someone who stole from me and give credit to someone else say, “I got it from them, and they are a big developer, so you kanged them” – when I had the feature in the hands of end users three to four months prior in this specific example.

It seems people also refuse to look at date and time stamps, as well as actual discussion of the work taking place in my threads days to months before someone else invented the same modification.

Where does someone start if they want to do what you do?

I would find a developer who commits their changes to a public repo, and describes what the commit (update/change) is actually doing. Some develoeprs just say “updating,” so if someone is attempting to learn from that developer, it’s not gonna happen.

We have a public version of my Shooter/Shooter_u kernel. It does not have any of my “written from scratch” work, and most of the patches are just common sense changes. If you were to look at the description of the commit, I am very detailed in what the change does. People should be able to learn from git repos like this.

What do you enjoy doing that’s completely not development related?

I like working on cars, watching Adult Swim, and just hanging out. I also fix computers as a hobby, and I have a 1/8 scale on-road gas powered RC car that does about 75MPH … just gotta find a new place to run it.

Everything else is kinda development related – doing new things, seeing how I can improve my current work.

I managed to miss the cease fire between you and another developer. Has the old XDA drama finally died down as a result of that, or does it still go on?

There is still drama. Other developers trash talk on Google+ or Twitter. But the trash talking on XDA itself seems to have died down.

I’m still accused of kanging, and people are still upset I don’t release source. Once in a while (mainly in the HTC Sensation forums), people out of no where start accusing me of software piracy and distributing warez due to the fact I do not release source.

I just wish XDA’s moderation team would educate these users on ALL aspects of the GPL. General users assume GPL means source is posted. There is a lot more to it that that.

Looking at GPL’s requirements, I would say less than 5% of kernels on XDA are 100% compliant, and in the eyes of the law, you are either compliant or you are not. I think if the general public had a better understanding of ALL of the GPL requirements, they would back off.

What’s your current motivation in making these kernels – has it paid off for you?

I still mainly make them for me. Everything I build, I run. There are a few exceptions. My 8660 series kernels are universal, using the same source I use to build for the HTC EVO 3D, I can build for the Sensation, GSM EVO 3D, Rezound, Amaze, and Vivid devices. I made patches similar to this that allow one source tree to work on different devices depending on what default configuration file you pick.

That specific patch there allows Buttered Toast to work on GSM and CDMA devices, so any changes made to non-board specific files apply to both devices. Using patches like that, my kernels support many devices, so I build for the Sensation, GSM, and CDMA as it really requires no additional work from me and there is a high demand for them.

I also build for the EVO 4G LTE, as I now have one of these. To be honest, if I did not own a device, or use my own work, I would not build for it. No matter how large the demand, due to the amount of back stabbing and drama, the pay off is proving to myself as well as the community that I can do things without source.

I had fixed the GB HTC EVO camera issue for the 2.17 EVO 3D base before HTC released source for the 2.17 OTA, I had stable and working WiMAX in the Linux 3 kernels for the 3D long before HTC released source; where other developers had WiMAX, it was not stable, and caused reboots.

You also could not turn them on, then off, then back on. My WiMAX builds were as stable as the HTC builds are today.

Has restricting your kernel to members only limited the base, or do you find that people are flocking to your site to get it?

I never asked for donations, and actually refused donations for the longest time, so I’m not sure. I also don’t track downloads.

When I did release on XDA, I was average 3000+ downloads of each build. I do not have this many active members on my site, so the user base is most likely reduced. I also know people are unaware I even still develop. I get a lot of “wow, I found you,” PMs on my site.

Without naming names, could you tell us a bit about how some people are getting credit for stolen kernel work?

I have several git repos located on bitbucket, and one day, I created a new one, and uploaded some source to it, but forgot to make it private. This source was 100% public for about two days.

I got an email from someone thanking me for making my work public. At this point I realized the error, and fixed it. This “leaked” source contained my gingerbread kernel source, and a few branches within the repo for Linux 3.0.8 and 3.0.16.

Some of the tweaks and modifications to this source are in use today all over XDA, and others are claiming credit for it. Since I do not make my work public, and I am no longer on XDA, the general users have no idea who truly had what first. What I will say is there were features in my Gingerbread kernels three-four months before others had them.

I have also helped other developers on G-Talk for hours, giving them fixes, and sharing source with them to help them get started. These developers are active on xda-developers today, and they are claiming credit for work I produced, and they have the nerve to tell the community I kanged them, yet – prior to me teaching them how to even compile a kernel – one of the two developers had no clue.

It comes down to money. I never asked for donations, and I actually refused donations for several months. Other developers are claiming credit for my work, and asking for donations from their users.

I am not the only victim of this. I recently discovered a well known developer is pulling patches from either kernel.org, CAF, or Linaro, applying the patch as it existed upstream, then claiming they had to modify it for HTC, or modify it for Linux 3.0. In reality, the patch is 100% line for line identical to the upstream patch.

Then these developers openly accuse me of kanging this same exact patch from them, when in fact I got it from CAF or Linaro (or kernel.org). If you look at the change log in my OPs, you can see each change has credit to someone. I do give credit to the person or place I got the change from, but this may or may not be the person who created the patch or modification. (I just don’t have the time to research where something really came from.)

Is there any way for an average user to tell when someone’s kanging kernel components?

The easiest way is to watch the developers, and see who releases what first. Watch what features are being discussed in the threads. For example, a user requested a specific feature on my forum. They wanted programable start and end points for the sweep2wake mod. I told them I would have it soon. Within the week, I released it, and asked for feedback.

The day after people started providing feedback for it, another developer came up with the same tweak to sweep2wake, but his was just start point, not end point. This developer then told everyone he invented it, and it was his original idea he came up with on his own.

The next day, he added code to allow the end point to be configured. Since I am not on XDA, users there believe this other developer truly invented this, due to the fact they are 100% unaware it was in the hands of my users days prior. I also had this repo public until this other developer claimed credit for my work, so a developer could get the patch for Buttered Toast.

Another way to check for kangs is to look at KMSG logs. In the past i had put code in various parts of the kernel that would display “ANTHRAX” in the KMSG logs, but when you searched the source code, you would not see anthrax in plain text.

When I did this in a public repo, there are at least three other developers who had “anthrax” in their KMSG logs, yet they denied kanging me, and claimed I kanged them. The developers have caught onto this recently, and the ones who did have “anthrax” in their KMSG logs got it removed within a few hours – but they still got caught. Most users do not know how to read source code, or download github repos, but I have “easter eggs” hidden in parts of my code, even in Buttered Toast.

Have you ever considered measures that an end-user could see to protect your kernel code such as a 4-fingered downward swipe would display the AnthraX logo Easter egg? That way everyone would know what the deal was and not just developers.

Stuff like this would be more userspace (ROM-side) mods. I managed to hide KMSG messages in open source versions of my kernel. The best protection is not releasing source. I have become super aware of the status of my bitbucket repos now, and I double check new ones before I upload the source.

Who are some other developers or what are some new developments that are out that you think people should be aware of?

Just like I had two kernels in the past – one for XDA with full source disclosed, and one for InfectedROM without source disclosed – ButteredToast is the equivalent of my GPL compliant kernels of the past. So, on XDA, these would be the kernel of choice for GSM and CDMA EVO 3D devices. They do not contain the audio drivers, Bluetooth drivers, the SMP driver, and CPU tables and drivers I use in Anthrax, but it is the closest you can get to the original. There are other “un-authorized” copies of anthrax/Buttered Toast out there – but users should support the “original” and not wanna-be copies.

What do you hope for the future of development? Of XDA? Of AnthraX-Kernels.us?

I will personally continue to develop on devices I personally use – and provide a drama-free place for other developers to share their work.

As far as XDA, it is what it is. It was my choice to leave there – not theirs. The only downside to me not being on XDA is that the general user base is 100% unaware that I still work on kernels, and this kinda makes it hard to prevent the kanging going on.

If my kernels were still on XDA, people would see that I did in fact have the features I’m accused of kanging long before other developers.

As far as anthrax-kernels.us – it is a drama free place – anyone is welcome to join. First sign of drama, and you’re gone.

*************************

If you have any questions for Chad, feel free to post them below.