Perl 6 lists, episode 1

This past week I’ve worked on adding list support to nom, including lazy lists and infinite lists. Lists in Perl 6 have had a bit of a convoluted history; early drafts were very non-specific about how Lists should work, and early designs tended to run into problems during implementation, leading to many redesigns of the spec.

Rakudo has gone through at least five different re-implementations of lists and arrays; the latest occured in Summer 2010 in preparation for the Rakudo Star release. Lists are extremely core to all of Perl 6, so small changes tend to impact the entire codebase. Also, features such as laziness and infinite lists invalidate many assumptions we Perl programmers have about working with lists and arrays. For example, asking a list for its size (i.e., +@a) generally destroys any laziness it might have had, as it has to go and generate all of its elements. Similarly, storing a lazy list inside of an array loses laziness, since array assignment is eager in Perl 6.

(For those who are wondering: array assignment defaults to eager evaluation because otherwise you get all sorts of surprising and non-Perlish action-at-a-distance effects.)

Similarly, passing I/O objects to various list functions would end up consuming the I/O stream entirely, such that the contents were lost to the caller. But you don’t want to adopt a model where you preserve every element all of the time, because then filters such as map and grep fail to act like pipelines and become very expensive in terms of memory consumption.

So, the Perl 6 specification and implementations have been through quite a few false starts before landing on a lists API that seems to work. We now have Lists, Arrays, Parcels, Captures, LoLs, Nil, Scalars, Seqs, flattening, interpolation, sinks, “mostly eager”, “mostly lazy”, and a bunch of other things that are used behind-the-scenes to get things to “just work” for the typical Perl 6 programmer. If seeing all of these new types frightens you, then perhaps a physics analogy helps: many of the details are like quantum physics to chemists — the Perl 6 implementors have to understand the quantum physics, while Perl 6 programmers can do a lot of useful chemistry without having to be aware of the gory details underneath.

Rakudo (master) contains the “first-draft” implementation of the latest lists API, developed last summer. Its emphasis was more on “get something working for Rakudo Star” than “make something elegant and efficient”. The key insight that enabled us to finally get something working is that iterators have to act immutable. More on this in a later post.

My original plan was to go back and revisit lists in Rakudo master, but with the “nom” branch making as much progress as it has (over 500 commits in the last 30 days!), I’ve decided to work on it there first. Nom’s new object and a ton of other internal improvements make the implementation a lot easier. In fact, unlike the previous implementation of lists, this new one is being written almost entirely in Perl 6. For efficiency reasons we may later convert parts of the newimplementation back to PIR, but we’ll do that only after we have a working P6 implementation first and we can benchmark it to see where the bottlenecks are.

Thus far the new implementation is working out extremely well. Because it’s in Perl 6, and because we now have far better object and container models than we previously had, it’s a much cleaner and more elegant implementation than before. So far it’s still slower than Rakudo master’s lists (which are already horribly slow), but it is also much easier to inspect and optimize than it was previously. For example, yesterday I was able to hotpath the code for generating numeric ranges such that it’s now several orders of magnitude faster than what we have in the master branch. Jonathan was able to profile the code and found a memory leak and another 30% improvement. So, we expect great increases in performance very soon. As far as capability, nom’s lists already handle some features that Rakudo master doesn’t get right yet.

I have a pending Hague Grant to write up the documentation for lists in Perl 6, primarily Synopsis 7. (The current Synopsis 7 draft is at least two iterations out of date with respect to the current design, which confuses a lot of people.) Over the next week or so I’ll finish up the implementation in nom, then write up the results as the new Synopsis 7 and make edits to the other synopses as appropriate. I’ll mark progress here on the blog and provide some examples of the various concepts as that progresses.

If you’re just dying to see what we have now, the source code for Lists and the other builtin types is available from the nom repository on github, in https://github.com/rakudo/rakudo/tree/nom/src/core .

Pm

Posted in Uncategorized | 2 Comments

Rakudo (nom) reaches milestone — no more .pir files!

As Jonathan Worthington and I have posted in various articles, we’re working on a new implementation of Rakudo Perl 6 (the nom branch) that is based on a new meta-object model instead of the object system provided by Parrot.

Today we reached a nice milestone of sorts, as I removed the last “.pir” file from the Rakudo-nom source tree. This means that all of the source code files in the Rakudo repository are now “Perl 6”. (Some of the files are actually in NQP, which is a Perl 6 lite of sorts. See this article for more details.)

To be certain, there are still some PIR features embedded in some of the source files — either using pir::opcode or the old Q:PIR syntax. However, we’re steadily eliminating any of this Q:PIR code that is being used, and the pir:: opcodes will ultimately replaced with abstract nqp:: operations that can be more universal than just Parrot.

Otherwise Rakudo-nom is progressing nicely, within the past week we’ve been restoring and updating a lot of the core Perl 6 features. We already have multisubs, basic list and hash operations, basic parametric roles, and the like. We’re also stubbing in many features that have been missing from the master branch, such as Stringy and LoL.

The nom branch isn’t yet at a point where it can compile Test.pm and pass tests, but at the speed things are moving it won’t be long before we are. I can say that developing the core Perl 6 compiler code on the new object model is far more straightforward and easier than it was under the previous Parrot object system. Credit goes to Jonathan and the others who helped him for this — it’s a huge improvement. We also hope to see some dramatic speed improvements over Rakudo master, as well as vastly improved optimization possibilities.

We don’t have a solid projection for when the nom branch will replace the current Rakudo master, but I expect it to happen well before the end of this summer.

In an upcoming post I’ll describe all of the old-and-new terms that have been occurring in the Rakudo universe (“6model”, “nom”, “nqp”, “master”, “nqp-rx”, “Rakudo Star”, “parrot”, “zavolaj”, etc.), to give a better idea of how this all fits together. Stay tuned!

Pm

Posted in perl6, rakudo | Leave a comment

Review: Samsung Galaxy Tab Leather Notebook Case

For weeks after purchasing my Samsung Galaxy Tab I had been looking for a good case for it (ideally leather). None of the cases I found seemed worth buying — either the customer reviews indicated the case was of poor quality, or it had obvious shortcomings. For example, I want a case that is thin and opens from the left side of the Tab, so it can double as a stand.

Recently I came across the Galaxy Tab Leather Notebook Case in Samsung’s Marketplace and purchased one for the Tab. I like it a lot. This case seems surprisingly difficult to find on the web — few online stores carry it and even the Samsung marketplace doesn’t make it easy to find. Since I haven’t seen many other reviews or mention of this case, I figured I’d write one on my own experiences.

At $49.99 in the Samsung Marketplace, this case is definitely more expensive than many other leather cases that are available. And although it’s a “Leather Notebook Case”, in fact only the outside of the front cover is leather–the back cover is plastic. I knew when I purchased the case that there was a plastic shell involved to snap on the Tab and protect the corners, but I expected the entire outside of the case to be covered in leather and was disappointed that it doesn’t. For fifty bucks I really expected something a bit more upscale than a back of exposed plastic.

Samsung also seems to want to put their name all over the case- each surface, both inside and out, has the word “Samsung” on it somewhere.

But other than the exposed plastic and the high price, I really like the case much better than any other cases I’ve yet come across for the Tab. The case itself is very thin, so it doesn’t add a lot of bulk or thickness to the Tab. The plastic back cover snaps on quite securely, yet doesn’t interfere at all with the display surface. While using the Tab I don’t even notice the case is there unless I stop to think about it.

An important feature for me is that unlike many other cases I’ve seen, this case opens from the left and thus can be used as a stand for the Tab while leaving all of the buttons in the correct orientation. Many cases I’ve seen open from the right, which puts the power and volume buttons on the bottom and the android buttons to the left of the display, which is completely wrong to me. Thankfully this case gets it right — power and volume on top, android buttons to the right.

I was surprised (and pleased) to discover that the case has small rubber pads on its edges that keep the Tab from sliding on a smooth surface when used as a vertical stand. These pads are much stronger than they appear; I initially expected the Tab to have a tendency to slide away from me while using the touch screen, but unless I push really forcefully it doesn’t move at all. This often comes in handy for reading ebooks or using the Tab for games; I can set it upright on the table in front of me and reduce strain on my neck and arms.

The interior of the case is lined with velvet or something similar, good protection for the Tab’s screen and back when the case is closed. There aren’t any pockets or anything like that — it’s just a good, slim sturdy case.

Overall the case makes me feel much more comfortable about transporting the Tab from place to place without getting dinged or scratched, and it doesn’t add a lot of bulk or weight. Given the lack of leather on the back I think this case really ought to be priced around $35 instead of $50, but overall I’m quite happy with my purchase. I definitely recommend it to others who might be looking for a case for the Galaxy Tab.

Update 2011-04-15: Robert asks in the comments if the cover folds back flush with the case when open. It folds back very nicely, as you can see in the photo at right. It’s not perfectly “flush” because the back has a slight curve to it, but I certainly don’t find the cover to be at all in the way when it’s folded back like this.

Posted in android, review | Tagged , | 4 Comments

New NQP repository, new “nom” Rakudo branch

As mentioned in the last week’s NQP Roadmap, we’ve now broken out the new 6model-based NQP into its own repository on GitHub: http://github.com/perl6/nqp . We know there may be a bit of confusion between the two versions; just remember that “nqp-rx” always refers to the older one and “nqp” without the -rx refers to the new one.

Today Jonathan also created the “nom” branch in the Rakudo repository where we’ll be converting Rakudo over to using the new NQP. We expect this process to take several weeks. Rakudo compiler releases will continue to be made from the existing “master” branch until the new branch is ready.

Our hope and expectation is that the April 2011 Rakudo Star release will be using the new 6model implementation. Stay tuned!

Pm

Posted in nqp, parrot, perl6, rakudo | 5 Comments

Incognito FTW

In another “obvious when you think about it” moment, yesterday I realized that Google Chrome’sincognito mode” is a quick-and-easy way to see how a website or url will be seen by an anonymous user (i.e., not me) without having to log out. Firefox’sprivate browsing mode” can be used the same way.

For example, many times I will send a url of a Picasa web album or other page to friends and family, and I’d like to test the url before sending it. However, since I’m usually logged in to the site at the time, what I see at the url is often different from what they will see. Previously I would sign out from the site in order to test things (and then sign back in later); with “incognito mode” I can view the site as an anonymous browser without having to sign out.

I already have my F12 key mapped as a hotkey to open any highlighted url in a new browser tab (very convenient); I’ve now mapped Ctrl+Shift+F12 so that it opens a highlighted url in a new incognito window:

google-chrome --incognito --new-window $(xclip -o)

So now when I want to test how a url will be viewed by others: highlight the url, press Ctrl+Shift+F12, and that’s it!

Pm

Posted in linux | Tagged , , , | Leave a comment

Adding USB ports to my monitor

The USB ports built into my monitor died a few years ago, forcing me to start using the USB ports on my desktop tower (which is often much less convenient to reach than the monitor).  I looked for replacement monitors with USB ports, but they weren’t in my price range or didn’t have as good a video quality as the monitor I had.

Then one day it occurred to me: Just buy a small USB hub and stick it directly to the monitor.  This is probably obvious to many people reading this, but I never saw anything like it on the web (and wished I had), so maybe this article will help someone else.  Here’s what I ended up with:

All I did was get an inexpensive 4-port USB hub (around $10 at Fry’s), use a 3M Command Poster Strip to attach it to the bottom corner of the monitor, and voila — handy USB ports again! The particular hub I chose has one of the USB ports at the back, so I also got a 1-foot USB cable and stuck a USB multicard reader next to it as well.


When I was shopping for a new widescreen monitor last month, I didn’t have to look for USB ports builtin to the monitor, since I knew I could always pop the hub off my old monitor and stick it on the new one. Again, this is nothing particularly clever, but it has turned out to be a real convenience for me.

Pm

Posted in projects | Tagged | 2 Comments

NQP Roadmap 2011.01

Over the next couple of weeks we will be rolling out a new version of NQP that is based on new backend code for object management and multisub dispatch. This document describes the changes we’ve identified for the new version of NQP, the expected impact on the existing Parrot/Rakudo/NQP ecosystems, and our expected timelines for the changes taking place.

Most importantly, the new NQP contains a shift in philosophy from previous versions. Previous versions (including “nqp-rx”) aimed to avoid any runtime component whatsoever, providing simply a HLL interface to whatever capabilities were provided by the underlying VM (i.e., Parrot). The new NQP now explicitly expects some small amounts of “library code” to be loaded on top of the VM, used to provide a basic framework for object management, multidispatch, and regexes. However, the overall goal of NQP continues to be a very minimal Perl 6 syntax for writing code libraries and compilers for VM backends such as Parrot.

And yes, the new NQP is explicitly designed to enable other languages implemented with NQP to run on multiple VM backends such as the CLR, JVM, etc. Several key people are already making important contributions in this area now, and we expect more in the months ahead. But for the next few months our focus will be more on improving Rakudo’s performance and faithfulness to the Perl 6 specification over trying to make it run on multiple platforms. (Of course, if we quickly discover that other VM backends are likely to give significant performance benefits sooner, we may increase our efforts in this area.)

To simplify the transition, the new NQP will be hosted in a new GitHub repository separate from the existing perl6/nqp-rx repository. This should enable the old nqp-rx line to be maintained and available for those systems that depend on it, while allowing the radical changes to take place in the new one. In order to (hopefully) avoid confusion, from now on we’ll consistently use “nqp-rx” to refer to the existing implementation, and “nqp” to refer to the new one.

The new nqp implementation provides its own object metamodel implementation (commonly known as “6model”), avoiding almost entirely the object framework currently provided by Parrot. In other words, the new nqp does not make use of Parrot’s Class and Object PMC types, nor does it use the P6object interface library. Both the Rakudo and Parrot development teams have identified in the past that Parrot’s object system needs some significant work; it’s our hope that nqp’s implementation will be able to jumpstart this for Parrot, perhaps by adopting it into its core in some manner. The OO framework in nqp has been explicitly designed to be flexible enough that other languages can use it to implement their metamodel as well. It makes very few assumptions at its core, and tries to create a level playing field in terms of access to performance-improving features needed by languages other than Perl 6.

The new nqp also contains a replacement implementation of multiple dispatch — it no longer makes use of the Parrot MultiSub PMC type. The new dispatcher is more efficient and truer to the Perl 6 semantics (and hopefully flexible enough to easily support or implement dispatch semantics for other languages as well).

As a result of these changes, nqp will also be embarking on updated implementations of significant portions of the Parrot Compiler Toolkit (PCT), to take advantage of the optimization possibilities inherent in the new object and multidispatch implementations. In fact, we expect to (re-)implement PCT as nqp source code in the nqp repository, borrowing liberally from bacek’s pioneering efforts in this area. The new PCT will also likely obtain a new name (yet to be determined), to avoid confusion with the existing “Parrot Compiler Toolkit”. Existing references to Q:PIR, pir::opcode(), :inline(‘…’) and the like will be phased out in favor of nqp::* generic equivalents.

Concurrently with the above changes, we will be migrating Rakudo to use the new nqp as its underlying implementation platform. This will be done as a branch of the Rakudo repository, much like the “alpha->ng” migration that took place in late 2009/early 2010. This migration is expected to be completed in the next 2-3 months. In the meantime, the Rakudo “master” branch will continue to target the existing nqp-rx and Parrot systems. The next Rakudo Star release is not scheduled until April 2011 — as the new Rakudo branch nears completion we will determine how it will impact the timing and content of the next Rakudo Star release.

We expect that HLL translators and programs that used nqp-rx primarily as a high-level implementation language will be able to migrate to the new nqp with only minimal changes (and hopefully substantial performance improvements). Grammars shouldn’t require any significant re-work, and much of the action methods and PAST construction should remain the same as before. But the devil for this will of course end up in the details, which we’ll deal with as we encounter them.

Parrot libraries and subsystems that are using nqp-rx to access specific Parrot features will likely have more difficulty migrating to the new nqp, as nqp abstracts away from the low-level Parrot details a bit further. Our plan recognizes and fully understands that Parrot may elect to neither provide nor support nqp directly in its distributions, and may even migrate existing tools and libraries completely away from the existing nqp-rx and PCT. Or, Parrot might decide to embrace NQP more fully to take advantage of its new optimization and compiler toolkit capabilities. We can likely find ways to preserve the ability for NQP to access low-level Parrot features for Parrot-specific libraries. (This becomes far more conceivable if Parrot adopts the new object metamodel, which seems likely at this point and is the major impetus behind the changes to nqp.) Or, the existing nqp-rx could continue to live on as its own project targeting Parrot subsystems, perhaps even being adopted into the Parrot repository. Whatever the Parrot team chooses to do in this area, nqp will support as best it can within the goals and plans described above.

In particular, there is no expectation or request that the new nqp will have to be bundled with Parrot in order to support Rakudo. The new Rakudo branch will either pull or bundle its desired version of nqp directly from the nqp repository, analogous to the way that the current Rakudo master branch pulls from the Parrot repository as specified by PARROT_REVISION.

We recognize this new NQP implementation will results in significant changes to the existing Rakudo/Parrot/NQP ecosystems, but we’re expecting that the short-term pains will be far outweighed by the performance and capability improvements the new design will offer. We also know that the new implementations offer plenty of new possibilities for -Ofun for many of us, and we hope that many other people will want to join in the -Ofun as we explore the new frontiers this makes available.

Posted in nqp, parrot, perl6 | 5 Comments