After a period of testing and evaluating continuous integration products over the past few weeks, I have selected Hudson as the replacement for Buildbot.

Hudson was chosen for a variety of reasons, including the fact that it is cross platform, and that with a bare minimum of effort, you, the users can help run build systems to assist with porting Shinka to your systems.

Presently Hudson is configured to build on 4 buildslaves hosted by Yagisan, those being Linux amd64 GCC 4.4, Linux amd64 Intel Compiler 11.0, Linux amd64 LLVM/Clang, and Linux amd64 Sun Studio 12. Due to the increased memory requirements of Hudson, and the Java virtual machine it uses, I have had to limit this to a subset of all the build slaves I would like to host on this server. For more build slaves to be hosted on the server with acceptable performance a server RAM upgrade will be required - your donations can help realise this.

You can see the real time results of the builds located in the timeline - at present only debug builds are done at checking, although it is envisaged that other builds will be added as time goes by.

For now, anonymous users are permitted to start builds to see the current status of shinka on their platform, although this permission will be revoked if the build slaves are abused.

Due to poor performance, and irrelevant advertisements, infolinks inline advertisements have been remove from this site.

As you may been aware, we've been offline for about 2 weeks now. Unfortunately the motherboard in the server died, and a warranty replacement part wasn't available to be installed until tonight.

Apart from the obvious effects that two weeks offline can bring, it also interrupted testing of a replacement for the buildbots system. The system under test now has really demonstrated that the server needs more ram. I'd like to encourage all users to donate so that I may purchase an additional 8GB of DDR2 ram for the server ( ~$500AU at time of posting ). The existing 4GB of ram is extremely taxed even utilising kernel samepage merging to reduce virtual machine memory usage.

Due to an inablity to recover from the occasional network issues (typically dropouts during builds) to my fully virtualised build farm, the buildbots have been retired from service. Several alternatives are now under consideration to replace them.

I've just finished filtering the git repository to remove all doom64 data files. This required the removal of 1040 files, over a span of 3178 commits. These needed to be removed as they are not under a compatible opensource license.

As a result the entire repository was rewritten. If you have your own private development branches you will need to rebase. If you are just an end user - delete your shinka checkout and re-clone. You can clone from:

Mirror 1 (fast): git clone  git://gitorious.org/shinka/shinka.git

Mirror 2 (fast): git clone  git://github.com/Yagisan/Shinka.git

Master (slow): git clone  git://shinka.bpa.nu/git/shinka.git

I highly recommend using the mirrors, as they are updated within 30 seconds of a push to the master repository.

A git clone will now require around 130MB of disk space. The resources from doomsday for doom, heretic, hexen, and (the now removed) snowberry in the repository history are under the gpl v2 or later, so I have not filtered them from the repository.

It's been a while since my last post, and as evidenced by the lack of updates, I've not had much time to work on Shinka.

In the last few days I've made a few changes to the site, the three biggest ones are the removal of Gitweb and using an integrated git plugin to browse the source, a doxygen generated cross referenced source and the main thrust of this post - the buildbots.

Over the last week I have set up on this server several buildbots that are viewable via the buildbot tab above. What was expected to be a simple 1 hour job, however turned out to be not so simple.

It was initially planned that there would be 20 buildbots running, covering i386 and amd64 architectures, and a variety of operating systems and compilers. It quickly became apparent that even with kernel samepage merging active that the 4GB of ram in the server would not support 20 buildbots, and that 9 to 10 would be a practical limit.

It was decided then that due to this ram shortage (donations gladly accepted to increase the RAM to a maximum of 16GB - see link in the page footer) I would attempt to get at least one of every major target platform up and running. Linux and kFreebsd worked immediately, but opensolaris failed to install. I later discovered that the current builds of opensolaris have problems with Phenom CPU's and won't install. Naturally I have a Phenom CPU in the server so opensolaris will be delayed until this bug is fixed.

Next I tried Windows. They installed well, I got the builds running manually, we started a buildbot and ran a few test builds. Then a few more test builds. And a few more. In total 20 test builds. All displayed the same peculiar behavior. All succeeded in performing a git clone, and then the buildbot would hang, time out, and then report connection lost. Eventually I gave up on them, as I know it can git clone, but the point of a buildbot is that it builds and they didn't even get to the Cmake stage.

Next up was configuring builds on check in. I followed the instructions in the buildbot manual, installed the git hook, and ran a test commit. Nothing happened. Ran another test commit. Nothing happened. Gave up on it for now and ended up setting a 3 hour timer between builds. Thankfully the timer works.

So what's left to do ? I need the builds to trigger each time a push to the master repo is done. I need a try-build setup so I can test patches before they get pushed. I need working Windows 32bit and 64bit builders. I need Solaris, *BSD and OSX builders, and I need more RAM for the server if I plan to self host them. I want the buildbots to report coverage testing, static analysis reports, generate and upload doxygen generated documentation. Anyone familar with git and buildbot feel like lending a hand ?

Today I'd like to talk about advertising. Those of you without ad blocking filters on will have notice I am running both google adsense and infolinks inline advertisements on the Shinka website.

It has been an interesting (and unprofitable) exercise so far, as many of the advertisements are either not relevant, or public service advertisements. No, this post isn't about me deciding to remove the ad systems, but I will be making some changes to them - such as having donation banners appear every time google wants to display a PSA. To be fair, google adsense is context sensitive, and there isn't too much content here for now, but hopefully that will change in the future.

In addition to the donation banners that will replace adsense PSA's, there will be additional donation banners displayed on pages - they will be as unobtrusive as possible, but they will be there.

As you can well imagine, there are some costs involved in working on an open source project like Shinka, and it is hoped that the advertising and donations can offset these costs. In particular, development for the Windows and OSX platforms is expensive with all the software licenses involved, and hardware is also quite expensive.

Some of the costs incurred so far have been a UPS for the WWW server, a complete replacement of one of the development machines (the Athlon64 with Geforce 6800LE machine is no longer with us as of yesterday) with one based on a new ATI iGPU have put a large dent in my limited funds, which prohibits me from purchasing OSX licenses, and Windows XP/Vista/7 x86 and x64 licenses for some time.

I hope that those of you blocking the advertisements on this site reconsider it, and that those of you that value the project consider donating (money, software, hardware, or your time - eg programming, documenting etc) to support future development.

Regards, Yagisan

Here we are some 12 days out before the next release, and how are things turning out so far ?

• Engine header breakup completed (excluding public API). No mis-compiles or crashes on load when finished.

At this stage, it does not appear that this will be completed on time. It has been discovered that the headers are in worse condition than I had suspected. The engine itself has had some partial reduction in convenience headers, but most of the work has been on creating missing headers and fixing missing and incorrect declarations.

• Engine headers made C and C++ compatible.

Finished.

• Initial engine legacy code unit tests created.

Started, but stalled due to the header issues above. The system will now build the unit test library on systems that do not have it installed.

• Engine converted to a library.

Completed. This appears to be miscompiled by GCC 4.3, however GCC 3.3 is known to build it correctly, as does Sun Studio, and the Intel 11.0 compilers.

• New C++ based loader executable created.

Completed. A basic shim loader is done.

• Remove support for Windows, OSX, and non-Linux *NIXes.

Completed.

• Portability cleanups.

In progress. An interesting issue was uncovered where it was discovered that the inherited Doomsday code redefined the "uint" type to be 32 bits in size on all platforms. "uint" however is also defined by system header files on *nix. The Doomsday definition was deleted to prevent redefinition of the system type. This still works on 32bit systems, but is untested on 64bit builds. It also necessitates a full search of the source to see where uint is used if it really expects an unsigned int, or a 32bit unsigned int.

• Re-Implement Linux port.

In progress. Presently this is the only port. It is unlikely that porting back to libltdl from the custom dylib loader will be completed before release day.

To further discourage spammers I have enabled a CAPTCHA system for registrations. I also went through the user list and deleted every account that looked like it may have been a spammer account. If your account was deleted, and it wasn't a spammer account, please recreate it.

Regards,

Yagisan

As some of you may be aware, a new branch has appeared on the public git repo containing the current work in progress for the current release.

In it we have the new C++ loader (presently just a shim), the inherited doomsday engine is now a shared library, and the engine headers are C/C++ compatible.

Unfortunately it's not all going according to plan, as the game will segmentation fault on load when built with GCC 4.3. Interestingly, GCC 3.3, LLVM Clang and the Intel 11.083 compilers however do build and run without major issues - Clang and GCC 3.3 appear fine, Intel compiler with graphical issues.

I'm not sure what is causing GCC 4.3 to mis-compile the engine at this stage, so far I have established that the change that causes it was introduced somewhere between the Doomsday 1.9.0-beta5.1 and the Doomsday 1.9.0-beta6.2 release (the point at which we last synced). For those that are curious, this is the commit that exposes the GCC 4.3 mis-compilation.

Unfortunately I have been unable to test with Sun Studio 12.1 as everything built with that compiler is segmentation faulting leading me to suspect I do not have it installed correctly.

At this point I can either expend more effort into determining what causes the GCC 4.3 mis-compilation, or start wrapping characterization tests around existing code (which I suspect will be a lot more difficult than it sounds due to lots of header interdependencies).

In any case GCC 4.3, GCC 3.3, LLVM Clang, ICC 11.083 and Sun Studio 12.1 are reporting a lot of warnings in the codebase that need to be investigated and rectified.

Feel free to checkout the new branch and start tackling any of the issues I've raised, or even just test with any other compilers you have.

Regards, Yagisan

Well it certainly didn't take very long for us to come to the attention of spammers. The initial spam attack caused the CIA.vc reporting system to flood #doom-tech in OFTC with all the not so welcome spam from Russia so they no longer report on Shinka.

This was a shame. The wiki was cleaned up, and the ip addresses of the offending spammers have been blocked at the router.

Flash forward a few weeks and we got hit with a flood of spam last night, but this time something different happened. Two thirds of it was rejected by the spam filter after that first successful attack, and I just finished deleting the rest.

The whole point of a wiki, forum and bug tracker is to encourage contributions - unfortunately due to spammers I already require registration to edit or post, and I would prefer to keep the wiki,forum and bug tracker as open as possible.

Presently I'm looking at possibly setting up a CAPTCHA system to make it harder for the spam bots to register - what are your thoughts ? Would a CAPTCHA system further discourage people from registering and contributing ?

Regards, Yagisan

Some of you may be wondering, what systems is Shinka developed on ? After all, that can be a good indicator of what it will run on, and what features it should support.

I develop Shinka on 5 systems.

The primary development system for shinka is an eeepc 1000h. yes - that's right - one of those dinky little low power eeepc's is the main development system. Is your system more powerful than that ? if so, great - you know Shinka will run fine on your system.

Without further ado, these are my development systems:

eeepc 1000h

homebrew pc 1

homebrew pc 2

homebrew pc 3

homebrew pc 4

As you can see, it's quite an eclectic mix. It covers both major CPU vendors, all three major GPU vendors, and runs from very obsolete to fairly recent hardware. There are systems that support GLSL, systems that support shader assembler, systems without hardware texture and lighting, systems with hardware texture and lighting, single core systems, multi-core systems, and smt systems. This wide mix should ensure Shinka can scale from the low-end to the high-end, which will greatly assist in my goal to make Shinka portable to almost everything.

As I look through the codebase, it struck me, that although it could at the time of forking run on Windows, Linux, and OSX, that the code itself isn't very portable.

In fact it creates shims to emulate Windows functionality on other platforms, and actually re-implements standard library functions, such as the dynamic loader, and most frustratingly has features that are only supported on the Windows platform and no other. It is riddled with the assumption that int long and pointers are all 32 bit, and most likely isn't big endian clean either. In short, I'm not too happy with it, and think it needs to be refactored.

I want as much as practically possible, one single codebase across all supported platforms. Ultimately what is supported comes down to a combination of two factors, firstly, do I, or any other developer have regular access to that platform, and secondly, do all our needed libraries and tools exist on that platform.

At present, we are not very portable. The Windows port never worked with cmake due to it using a non-standard linker trick with msvc which is yet to be fixed, and I don't have an OSX machine to test on either. That leaves just Linux as the only supported platform as that is what I am running. In a later post, I'll go through the libraries and tools we depend on, as well as the platforms available to me, and see what platforms we should run on without too much effort.

Late last night my time, I was contacted by The Doomsday Engine project (the DooM port we forked from long ago). They have been aware of this project and it's since November 2008 when I first started it.

Instead of the anticipated cooperative remote debugging I had expected, I received a strange request. They requested I change the projects name. Not once in our regular contacts since November 2008 did they express any concern about the name. This sudden and unexpected request certainly caught me off guard.

Their stated reason basically comes down to "We think people will confuse deng and deng-ng". As their users never refer to the project as deng, and they themselves didn't until recently, I didn't see any chance of confusion, but they were adamant the name should change.

They were not swayed by the argument that my chosen name honoured the project it had forked from, much like GZDooM and ZDooM, or Samaba and Samba-TNG.

Why are they suddenly so concerned over the name ? A few theories have been proposed such as; now that their website was deleted when their web hosting provider ceased hosting, that people googling for doomsday would instead find their way here, or that they are concerned that due to their long history of not making releases when promised, that our policy of quarter releases would have users migrate over as we do release when stated.

I'm not very happy with the request, but I will rename the project (and in doing so remove all references to doomsday) - even if I have no obligation to do so.

I feel that ultimately this request will hurt them more than it hurts me. At least with the name change I will lose any emotional baggage that is attached to the Doomsday Engine name. For now, the plugins will be renamed to the name of the game they are descended from for game plugins, or generic names if otherwise.

As for the engine, well - it's time to evolve into a a stronger, better port, and with that in mind, the new name of the engine will be: Shinka

How do you eat an elephant ? One bite at a time. Today I began eating that elephant, and it's time to chew like mad.

The refactoring of the doomsday headers has begun in earnest, de_network.h is no more and whatever included it, now just includes needed headers.

de_base.h however is taking a lot more time. This is a convoluted mess - during the breakup all sorts of interesting dependencies are appearing - such as b_device.c needing dd_wad.h to build, and files needing dd_zone.h building and linking - but with strange warnings ... diffing build logs to discover new warnings is not fun, especially with all the warnings we already have.

This refactoring is designed to help identify pinch points so I can start wrapping a test harness around deng-ng and begin behavior changing refactoring, and eventually convert to C++

The 20090601 release has been tagged and pushed to the git server.

This release has been a rather small release, with primarily only the rebranding patches and build system patches being pulled in for release.

The build system had an extensive overhaul, primarily targeted at GCC/Linux systems. It checks for, and enables if found, various security technologies in GCC including stack smashing protection, and fully relocatable binaries.

With these changes in place, support for other compilers can easily be added, such as the Intel or Sun compilers.

By default we now select the release with debugging info mode if a specific build is not selected, otherwise we have debug, release and minimize size release build options available.

Welcome to the official Shinka blog.

This blog will be used to communicate news, information and changes to, and about Shinka.

This blog is available for anonymous read access, however only registered users will be able to comment on posts.

Feedback is always welcome, so feel free to let us know what you think.

Regards, Yagisan