There are two items that stand out in my mind about this year’s conference:

1. Andrew Shitov and his crew are absolutely amazing at organizing and running a conference. This was the most flawlessly executed conference or event I think I’ve ever been to. Not only that, but Andrew and the other organizers made it look effortless, which to me is a mark of true greatness. I’m certain that in fact there was a lot of planning and effort behind it, but the entire team just looked relaxed and at ease throughout the event. I’d definitely encourage folks to attend any event that Andrew and this group organizes.

2. Riga is a stunningly beautiful place. I definitely want to return here again some day, and I’m grateful that the organizers chose this location.

Pm

As progress continued on the new “nom” branch of Rakudo since my last posting, it was becoming increasingly evident that regular expression support would end up being the next major blocker. I think we were all expecting that nom would initially use the same regular expression engine that nqp (and nqp-rx) have traditionally used. However, as I starting working on this, it began to look as though the amount of effort and frustration involved would end up being almost as large as what would be needed to make a cleaner implementation up front, and would leave a quite messy result.

So, last week I started on designing and implementing a new engine. Today I’m happy to report that nom is now using the new QRegex engine for its pattern matching, and that making a new engine was undoubtedly a far better choice than trying to patch in the old one in an ugly manner.

So far only nom’s runtime is using the new regex engine; the nqp and rakudo parsers are still using the older (slow) one, so I don’t have a good estimate of the speed improvement yet. The new engine still needs protoregexes and a couple of other features before it can be used in the compilers, and I hope to complete that work in the next couple of days. Then we’ll have a good idea about the relative speed of the new engine.

I’m expecting QRegex to be substantially faster than the old one, for a variety of reasons. First, it should make far fewer method calls than the old version, and method calls in Parrot can definitely be slow. As an example I did some profiling of the old engine a couple of weeks ago, and the “!mark_fail” method accounted for something like 60% or more of the overall method calls needed to perform the parse.

Qregex does its backtracking and other core operations more directly, without any method calls for backtracking. So I expect that this one change will reduce the number of method calls involved in parsing by almost a factor of 3. Other common operations have also eliminated the method call overhead of the previous engine.

The new engine also uses a fixed-width encoding format internally, which means that we no longer pay a performance penalty for matching on unicode utf-8 strings. This will also enable us to eventually use the engine to do matching on bytes and graphemes as well as codepoints.

I also found quite a few places where I could drastically reduce the number of GCables being created. In some cases the old engine would end up creating multiple GCables for static constants, the new engine avoids this. A couple of new opcodes will enable QRegex to do substring comparisons without having to create new STRING gcables, which should also be a dramatic improvement.

I’ve already prototyped some code (not yet committed) that will integrate a parallel-NFA and longest-token-matching (LTM) into QRegex, so we’ll see even more speed improvement.

And did I mention the new engine is implemented in NQP instead of PIR? (Although it definitely has a lot of PIR influence in the code generation, simply by virtue of what it currently has to do to generate running code.)

Ultimately I’m expecting the improvements already put into QRegex to make it at least two to three times faster than its predecessor, and once the NFA an LTM improvements are in it ought to be even faster than that. And I’ve already noted new places ripe for optimizations… but I’m going to wait for some new profiles before doing too much there.

Another key feature of the new engine is that the core component is now a NQP role instead of a class. This means that it’s fairly trivial for any HLL to make use of the engine and have it produce match objects that are “native” to the HLL’s type system, instead of having to be wrapped. The wrapping of match objects in the old version of Rakudo was always a source of bugs and problems, that we can now avoid. Credit goes to Jonathan Worthington for 6model, which enables QRegex to do this, and indeed the ability to implement the engine using roles was what ultimately convinced me to go this route.

While I’ve been working on regexes, Moritz Lenz, Will Coleda, Tadeusz Sośnierz, Solomon Foster, and others have continued to add features to enable nom to pass more of the spectest suite. As of this writing nom is at 244 test files and 7,047 tests… and that’s before we re-enable those tests that needed regex support. The addition of regexes to nom should unblock even more tests and features.

Some of the features added to nom since my previous post on July 2:
* Regexes
* Smart matching of lists, and other list/hash methods and functions
* Fixes to BEGIN handling and lexicals
* Implementation of nextsame, callsame, nextwith, callwith
* More introspection features
* Methods for object creation (.new, .bless, .BUILD, etc.)
* ‘is rw’ and return value type checking traits on routines
* Auto-generation of proto subs
* Junctions
* Backtraces

We’ve also done some detailed planning for releases that will transition Rakudo and Rakudo Star from the old compiler to the new one; I’ll be writing those plans up in another post in the next day or two.

Pm

We continue to see that nom performs much better than the previous version of Rakudo. Moritz Lenz added enough features to be able to run the mandelbrot fractal generator under nom, so we can compare speeds there. Under master a 201×201 set took 16 minutes 14 seconds to run, in nom it “naively” took 4.5 minutes, and with some further optimizations Moritz has it running in 3 minutes, for a factor five improvement over the existing master branch. And there are still many more compiler-level optimizations that remain to be worked on.

In the past couple of days I added the metaoperators back into nom. Furthermore, the new implementation is far more correct — metaoperators such as &infix:<X> and &infix:<Z> can now handle multiple list arguments instead of just two as in master. We still haven’t added back the hyperoperators; I plan to do that in the next couple of days.

Jonathan has been attending the Beijing Perl Workshop this week; his presentation slides are now available at http://jnthn.net/articles.shtml. Videos may be available soon. Even with his travels, Jonathan has continued to implement some of the needed lexical and role support in nom, so that we’re generally unblocked in making needed progress in the branch.

Carl Masak wrote a post introducing the Perl 6 type system; after reading an early draft of his post we discovered that several of the builtin types (Code, Attribute, Signature, Parameter) have been mistakenly implemented as subclasses of Cool. We’ve now fixed this in nom; we may or may not fix it in master.

Indeed, we’re already starting to phase out the master branch altogether. Yesterday I made a commit to master that effectively freezes it to always test against a specific revision of the spectests. This means we’re free to fudge and adapt the tests to the needs of the nom branch without concern for what it might do to testing in the master branch.

Speaking of tests, Moritz gave me some useful shortcut links for viewing different reports in our RT ticket queue. I’ve now set up a page on rakudo.org at http://rakudo.org/tickets/ where we can collect these report links and describe how the ticket queue works. One of the more useful links is http://rakudo.org/rt/testneeded ; this link shows a list of tickets that can be closed as soon as someone is able to confirm (or add) an appropriate test in the spectests. Writing tests is fairly easy, if you’re interested in helping with Perl 6 development, this can be a good place to start.

Here’s a summary list of features added to the nom branch since my last posting (five days ago):

* Complex numbers and numeric operator fixes
* Complex numbers have two native nums instead of Num objects
* Rat literals
* List.pop, List.reverse
* Initial LoL (list of lists) implementation
* map and grep
* Many string methods and functions
* metaoperators: Rop, Zop, Xop, !op, op=, [op], [\op]
* infix:<===>, infix:<eqv>
* Hash and Array hold Mu values, scalars default to Mu constraint
* Fixes to Configure.pl and –gen-parrot=branch
* Proper handling of $_, $!, and $/
* Improved exception handling and reporting
* Hash and List slices, including autotrim on infinite indices

In the next few days I plan to have regexes working in nom, finish off the metaoperators, and improve string-to-number conversions (including radix conversions).

For people looking to learn some Perl 6, to help others with learning Perl 6, or to do a bit of both, Bruce Gray has started “Flutter” — a suite of “micro-demonstration screens” for Perl 6. Essentially each screen introduces or demonstrates a Perl 6 feature or concept. Flutter is still in the embryonic stage, so it could use both content and implementation improvements and I’m sure that patches and pull requests will be extremely welcome.

We can still use help with triaging spectests and other tasks, if you’re interested in hacking on code or otherwise helping out, email us or find us on IRC freenode/#perl6. We can also use help with adding useful links and developer information to rakudo.org, if you’re inclined to do some of that.

I’ve also worked further on nom’s list implementation, and it’s now faster than lists and iteration in master. In fact, for loops in the nom branch now run about 80% faster than they did in the master branch.

We continue to eliminate PIR from the code base in nom. For the core setting, we’re down to 143 instances of ‘pir::’ and 22 instances of ‘Q:PIR’. The rest have been replaced by generic ‘nqp::’ opcodes that can someday be targeted to other virtual machine backends. Currently we’ve defined about 83 nqp:: opcodes that are used in implementing the core setting. For efficiency reasons we might not ever be able to eliminate all PIR from the core setting, but we should be able to get it to be small enough that it can be walled-off into VM-specific code files.

To give an idea of how fast things are moving — here’s a summary of the features that have been added to nom in the past seven days:

* fail()
* lexically scoped returns
* for-style loops and map, 80% faster than master
* better infinite lazy list handling
* gather/take
* try statements
* package-scoped variables, subs, and methods
* whatever currying
* Test.pm
* lots of builtin operators and methods
* dynamic variables, PROCESS and GLOBAL namespaces
* IO objects, including $*IN, $*OUT, $*ERR
* literal values in signatures
* quantified method dispatch (.?method, .+method, .*method)
* basic roles, including Associative, Positional, and Callable
* basic support for natively-typed lexicals (e.g., ‘int’, ‘str’, ‘num’)
* argument interpolation
* list assignment
* new say and .gist semantics
* magical string increment and decrement
* sequence operator
* series operator
* preliminary BEGIN/CHECK/INIT/END phasers
* smart matching (~~)
* inlined assignment

So, you can see things are active. We’re also in need of testers and people who can help us triage spectests and figure out what is causing them to not run. If you’re interested in hacking on code or helping with the tests — email us or find us on IRC freenode/#perl6!

Today we reached a nice milestone of sorts, as I removed the last “.pir” file from the Rakudo-nom source tree. This means that all of the source code files in the Rakudo repository are now “Perl 6″. (Some of the files are actually in NQP, which is a Perl 6 lite of sorts. See this article for more details.)

To be certain, there are still some PIR features embedded in some of the source files — either using pir::opcode or the old Q:PIR syntax. However, we’re steadily eliminating any of this Q:PIR code that is being used, and the pir:: opcodes will ultimately replaced with abstract nqp:: operations that can be more universal than just Parrot.

Otherwise Rakudo-nom is progressing nicely, within the past week we’ve been restoring and updating a lot of the core Perl 6 features. We already have multisubs, basic list and hash operations, basic parametric roles, and the like. We’re also stubbing in many features that have been missing from the master branch, such as Stringy and LoL.

The nom branch isn’t yet at a point where it can compile Test.pm and pass tests, but at the speed things are moving it won’t be long before we are. I can say that developing the core Perl 6 compiler code on the new object model is far more straightforward and easier than it was under the previous Parrot object system. Credit goes to Jonathan and the others who helped him for this — it’s a huge improvement. We also hope to see some dramatic speed improvements over Rakudo master, as well as vastly improved optimization possibilities.

We don’t have a solid projection for when the nom branch will replace the current Rakudo master, but I expect it to happen well before the end of this summer.

In an upcoming post I’ll describe all of the old-and-new terms that have been occurring in the Rakudo universe (“6model”, “nom”, “nqp”, “master”, “nqp-rx”, “Rakudo Star”, “parrot”, “zavolaj”, etc.), to give a better idea of how this all fits together. Stay tuned!

Pm

Today Jonathan also created the “nom” branch in the Rakudo repository where we’ll be converting Rakudo over to using the new NQP. We expect this process to take several weeks. Rakudo compiler releases will continue to be made from the existing “master” branch until the new branch is ready.

Our hope and expectation is that the April 2011 Rakudo Star release will be using the new 6model implementation. Stay tuned!

Pm

Most importantly, the new NQP contains a shift in philosophy from previous versions. Previous versions (including “nqp-rx”) aimed to avoid any runtime component whatsoever, providing simply a HLL interface to whatever capabilities were provided by the underlying VM (i.e., Parrot). The new NQP now explicitly expects some small amounts of “library code” to be loaded on top of the VM, used to provide a basic framework for object management, multidispatch, and regexes. However, the overall goal of NQP continues to be a very minimal Perl 6 syntax for writing code libraries and compilers for VM backends such as Parrot.

And yes, the new NQP is explicitly designed to enable other languages implemented with NQP to run on multiple VM backends such as the CLR, JVM, etc. Several key people are already making important contributions in this area now, and we expect more in the months ahead. But for the next few months our focus will be more on improving Rakudo’s performance and faithfulness to the Perl 6 specification over trying to make it run on multiple platforms. (Of course, if we quickly discover that other VM backends are likely to give significant performance benefits sooner, we may increase our efforts in this area.)

To simplify the transition, the new NQP will be hosted in a new GitHub repository separate from the existing perl6/nqp-rx repository. This should enable the old nqp-rx line to be maintained and available for those systems that depend on it, while allowing the radical changes to take place in the new one. In order to (hopefully) avoid confusion, from now on we’ll consistently use “nqp-rx” to refer to the existing implementation, and “nqp” to refer to the new one.

The new nqp implementation provides its own object metamodel implementation (commonly known as “6model”), avoiding almost entirely the object framework currently provided by Parrot. In other words, the new nqp does not make use of Parrot’s Class and Object PMC types, nor does it use the P6object interface library. Both the Rakudo and Parrot development teams have identified in the past that Parrot’s object system needs some significant work; it’s our hope that nqp’s implementation will be able to jumpstart this for Parrot, perhaps by adopting it into its core in some manner. The OO framework in nqp has been explicitly designed to be flexible enough that other languages can use it to implement their metamodel as well. It makes very few assumptions at its core, and tries to create a level playing field in terms of access to performance-improving features needed by languages other than Perl 6.

The new nqp also contains a replacement implementation of multiple dispatch — it no longer makes use of the Parrot MultiSub PMC type. The new dispatcher is more efficient and truer to the Perl 6 semantics (and hopefully flexible enough to easily support or implement dispatch semantics for other languages as well).

As a result of these changes, nqp will also be embarking on updated implementations of significant portions of the Parrot Compiler Toolkit (PCT), to take advantage of the optimization possibilities inherent in the new object and multidispatch implementations. In fact, we expect to (re-)implement PCT as nqp source code in the nqp repository, borrowing liberally from bacek’s pioneering efforts in this area. The new PCT will also likely obtain a new name (yet to be determined), to avoid confusion with the existing “Parrot Compiler Toolkit”. Existing references to Q:PIR, pir::opcode(), :inline(‘…’) and the like will be phased out in favor of nqp::* generic equivalents.

Concurrently with the above changes, we will be migrating Rakudo to use the new nqp as its underlying implementation platform. This will be done as a branch of the Rakudo repository, much like the “alpha->ng” migration that took place in late 2009/early 2010. This migration is expected to be completed in the next 2-3 months. In the meantime, the Rakudo “master” branch will continue to target the existing nqp-rx and Parrot systems. The next Rakudo Star release is not scheduled until April 2011 — as the new Rakudo branch nears completion we will determine how it will impact the timing and content of the next Rakudo Star release.

We expect that HLL translators and programs that used nqp-rx primarily as a high-level implementation language will be able to migrate to the new nqp with only minimal changes (and hopefully substantial performance improvements). Grammars shouldn’t require any significant re-work, and much of the action methods and PAST construction should remain the same as before. But the devil for this will of course end up in the details, which we’ll deal with as we encounter them.

Parrot libraries and subsystems that are using nqp-rx to access specific Parrot features will likely have more difficulty migrating to the new nqp, as nqp abstracts away from the low-level Parrot details a bit further. Our plan recognizes and fully understands that Parrot may elect to neither provide nor support nqp directly in its distributions, and may even migrate existing tools and libraries completely away from the existing nqp-rx and PCT. Or, Parrot might decide to embrace NQP more fully to take advantage of its new optimization and compiler toolkit capabilities. We can likely find ways to preserve the ability for NQP to access low-level Parrot features for Parrot-specific libraries. (This becomes far more conceivable if Parrot adopts the new object metamodel, which seems likely at this point and is the major impetus behind the changes to nqp.) Or, the existing nqp-rx could continue to live on as its own project targeting Parrot subsystems, perhaps even being adopted into the Parrot repository. Whatever the Parrot team chooses to do in this area, nqp will support as best it can within the goals and plans described above.

In particular, there is no expectation or request that the new nqp will have to be bundled with Parrot in order to support Rakudo. The new Rakudo branch will either pull or bundle its desired version of nqp directly from the nqp repository, analogous to the way that the current Rakudo master branch pulls from the Parrot repository as specified by PARROT_REVISION.

We recognize this new NQP implementation will results in significant changes to the existing Rakudo/Parrot/NQP ecosystems, but we’re expecting that the short-term pains will be far outweighed by the performance and capability improvements the new design will offer. We also know that the new implementations offer plenty of new possibilities for -Ofun for many of us, and we hope that many other people will want to join in the -Ofun as we explore the new frontiers this makes available.