Over the next couple of weeks we will be rolling out a new version of NQP that is based on new backend code for object management and multisub dispatch. This document describes the changes we’ve identified for the new version of NQP, the expected impact on the existing Parrot/Rakudo/NQP ecosystems, and our expected timelines for the changes taking place.
Most importantly, the new NQP contains a shift in philosophy from previous versions. Previous versions (including “nqp-rx”) aimed to avoid any runtime component whatsoever, providing simply a HLL interface to whatever capabilities were provided by the underlying VM (i.e., Parrot). The new NQP now explicitly expects some small amounts of “library code” to be loaded on top of the VM, used to provide a basic framework for object management, multidispatch, and regexes. However, the overall goal of NQP continues to be a very minimal Perl 6 syntax for writing code libraries and compilers for VM backends such as Parrot.
And yes, the new NQP is explicitly designed to enable other languages implemented with NQP to run on multiple VM backends such as the CLR, JVM, etc. Several key people are already making important contributions in this area now, and we expect more in the months ahead. But for the next few months our focus will be more on improving Rakudo’s performance and faithfulness to the Perl 6 specification over trying to make it run on multiple platforms. (Of course, if we quickly discover that other VM backends are likely to give significant performance benefits sooner, we may increase our efforts in this area.)
To simplify the transition, the new NQP will be hosted in a new GitHub repository separate from the existing perl6/nqp-rx repository. This should enable the old nqp-rx line to be maintained and available for those systems that depend on it, while allowing the radical changes to take place in the new one. In order to (hopefully) avoid confusion, from now on we’ll consistently use “nqp-rx” to refer to the existing implementation, and “nqp” to refer to the new one.
The new nqp implementation provides its own object metamodel implementation (commonly known as “6model”), avoiding almost entirely the object framework currently provided by Parrot. In other words, the new nqp does not make use of Parrot’s Class and Object PMC types, nor does it use the P6object interface library. Both the Rakudo and Parrot development teams have identified in the past that Parrot’s object system needs some significant work; it’s our hope that nqp’s implementation will be able to jumpstart this for Parrot, perhaps by adopting it into its core in some manner. The OO framework in nqp has been explicitly designed to be flexible enough that other languages can use it to implement their metamodel as well. It makes very few assumptions at its core, and tries to create a level playing field in terms of access to performance-improving features needed by languages other than Perl 6.
The new nqp also contains a replacement implementation of multiple dispatch — it no longer makes use of the Parrot MultiSub PMC type. The new dispatcher is more efficient and truer to the Perl 6 semantics (and hopefully flexible enough to easily support or implement dispatch semantics for other languages as well).
As a result of these changes, nqp will also be embarking on updated implementations of significant portions of the Parrot Compiler Toolkit (PCT), to take advantage of the optimization possibilities inherent in the new object and multidispatch implementations. In fact, we expect to (re-)implement PCT as nqp source code in the nqp repository, borrowing liberally from bacek’s pioneering efforts in this area. The new PCT will also likely obtain a new name (yet to be determined), to avoid confusion with the existing “Parrot Compiler Toolkit”. Existing references to Q:PIR, pir::opcode(), :inline(‘…’) and the like will be phased out in favor of nqp::* generic equivalents.
Concurrently with the above changes, we will be migrating Rakudo to use the new nqp as its underlying implementation platform. This will be done as a branch of the Rakudo repository, much like the “alpha->ng” migration that took place in late 2009/early 2010. This migration is expected to be completed in the next 2-3 months. In the meantime, the Rakudo “master” branch will continue to target the existing nqp-rx and Parrot systems. The next Rakudo Star release is not scheduled until April 2011 — as the new Rakudo branch nears completion we will determine how it will impact the timing and content of the next Rakudo Star release.
We expect that HLL translators and programs that used nqp-rx primarily as a high-level implementation language will be able to migrate to the new nqp with only minimal changes (and hopefully substantial performance improvements). Grammars shouldn’t require any significant re-work, and much of the action methods and PAST construction should remain the same as before. But the devil for this will of course end up in the details, which we’ll deal with as we encounter them.
Parrot libraries and subsystems that are using nqp-rx to access specific Parrot features will likely have more difficulty migrating to the new nqp, as nqp abstracts away from the low-level Parrot details a bit further. Our plan recognizes and fully understands that Parrot may elect to neither provide nor support nqp directly in its distributions, and may even migrate existing tools and libraries completely away from the existing nqp-rx and PCT. Or, Parrot might decide to embrace NQP more fully to take advantage of its new optimization and compiler toolkit capabilities. We can likely find ways to preserve the ability for NQP to access low-level Parrot features for Parrot-specific libraries. (This becomes far more conceivable if Parrot adopts the new object metamodel, which seems likely at this point and is the major impetus behind the changes to nqp.) Or, the existing nqp-rx could continue to live on as its own project targeting Parrot subsystems, perhaps even being adopted into the Parrot repository. Whatever the Parrot team chooses to do in this area, nqp will support as best it can within the goals and plans described above.
In particular, there is no expectation or request that the new nqp will have to be bundled with Parrot in order to support Rakudo. The new Rakudo branch will either pull or bundle its desired version of nqp directly from the nqp repository, analogous to the way that the current Rakudo master branch pulls from the Parrot repository as specified by PARROT_REVISION.
We recognize this new NQP implementation will results in significant changes to the existing Rakudo/Parrot/NQP ecosystems, but we’re expecting that the short-term pains will be far outweighed by the performance and capability improvements the new design will offer. We also know that the new implementations offer plenty of new possibilities for -Ofun for many of us, and we hope that many other people will want to join in the -Ofun as we explore the new frontiers this makes available.
Having just written an overview of PCT for a class, can I add a feature request for the NQP Compiler Tools? Documentation. PCT is reasonably documented, but I had to draw on examples and tutorials instead of API documentation to realize how to do things like declare variables and functions.
I know it’s an open source project, and I intend to help when I have time, but having the people writing the code keeping documentation as a goal really helps keep everything in sync. Examples are good, the fact that NQP and Rakudo uses them is better, but clear API documentation really makes getting others to adapt it much easier.
Documentation has always been one of our goals, but our tuit supply has been limited. I’m definitely planning to create a nqp tutorial and documentation as part of this effort.
I understand a lack of tuits. Graduate school is using up much of my own. But I’ll see abut expending some of my spare ones to help out. 🙂
Truly difficult to find competent individuals for this topic, nevertheless, you be understood as you no doubt know what you are preaching about! Thank you
Pingback: New NQP repository, new “nom” Rakudo branch | Pmthium