Creating an EJBQL query using a constructor expression in the SELECT clause

I'd read you could construct an entity summary dto direct from an EJBQL query. After a little fiddling I got it to work as follows:

public List listEntitySummaries() {
    String query 
            = "select new com.myapp.model.dto.MyEntitySummaryDTO(e.id, e.title) from MyEntity e";
        
    return (List) em.createQuery(query).getResultList();
}

The DTO in this case was a very simple java bean. I didn't need to declare it in persistence.xml. I did have to use the fully qualified classname however otherwise Toplink (in my case) couldn't find the constructor.

EJB3: Listing the Complete Contents of a Table

It took me a while to work this out. Here is is for posterity:

String queryString = "SELECT r FROM Recipe r WHERE r.id > 0";
Query myQuery = em.createQuery(queryString);
List result = myQuery.getResultList();

Note to Self: Windows Pathnames in Java

This gets me every time. I had to specify a jvm option in a maven properties file. It took ages, due to spaces in the path name. This worked:

maven.junit.jvmargs=-javaagent:C:\\Docume~1\\anlaw\\.maven\\repository\\javax.persistence\\jars\\toplink-essentials-agent.jar

Hopefully I don't forget again...

Maven 1.0.2: Adding a resource to a jar

Want to add a resource such as an xml config file to your generated jar in Maven 1.0.2?

1. Create a directory called ./resources in the base directory fo your project and place what you want to include in it (if you want to place it in a specific directory in the jar, such as META-INF, echo the dir strcuture you want in the jar)
   
2. Add the following to your POM:

<build>
<resources>
<resource>
<directory>${basedir}/resources</directory>
<includes>
<include>**/[file to include.xml]</include>
</includes>
</resource>
</resources>
</build>

Create your jar, and et voila!

Agile Development Ramblings: Part IV
The Documentation and Handover

Up to this point, we had been very successful. As far as the AU's were concerned, we had done a great job. They were all happy to sign off the finished product. However we had not built the final, deployable product. While not strictly speaking a true prototype (the vast majority of the front end code was planned to be kept rather than thrown away) everything behind our Session Facade was to be re engineered. This was to accommodate a wider and separately architected Strategic Portal Framework based on Websphere Portal and an SOA based Model / Persistence store.

In addition, this further development work was not to be conducted in-house and was instead to be handed to a development partner with little Agile experience. Further compounding this was the fact that further development was not scheduled to begin immediately. The original plan had been for us to verbally hand over as much information as possible. This was now impossible. Instead, we had one week to write it all down. To be fair, an awful lot already was. Indeed a great deal was already encoded in things like Maven. What we didn't have was anything about the architecture. Nothing up to date or accurate anyway.

The Agile Architect?
The point of Agile programming and development, I have learned, is that “the Code is the Blueprint”. Roughly translated, this means that everything you need to know about the resulting application is in the code; be this the code itself, the comments, the javadoc, or all the supporting (usually auto generated) xml and database schemas and properties files. This is all well and good, but to be able to understand it, you need to know Java and SQL. You also need a lot of spare time and better hope the developers write legible code.

This wasn't what the development partner was expecting, and so we had to go about reverse engineering things such as Logical Data Models, UML Diagrams, textual descriptions of the Architecture etc. This wasn't very Agile. We had been lucky in that we hadn't had a requirement to continuously produce these artifacts throughout the process. It would have hampered our progress considerably, always updating them and making sure they were accurate. However, it was clear why each and every one of them had been requested.

There is a current trend in the Agile world against the “Architect” and I agree that it is a grossly overloaded term. However, not everyone can, or should have to read code to see how something works. There is a requirement, even a need for there to be an intermediary between the Business Analysts and the Developers. There is a need for the production of logical and conceptual models which avoid implementation detail but provide enough data to inform strategic discussions as well as the nuts and bolts of building the thing.

A major benefit of Agile is that developers are plugged into the eventual users of their system, rather than being obscured from them by the analysts, architecture and architects. However, the current model of an architect is the “Big Up Front Design” / “Big Requirements Up Front Requirements” type of person which Agile has shown to be lacking. The old “develop to a paper plan and an interface” model Developer is changing, and I believe that Architects need to do the same; perhaps adopting a hybrid role like that which I played on this project will emerge – what I call the “Side On” approach (as opposed to the traditional “Top Down” or “Bottom Up” approaches). Then everyone has a relationship with the users. Perhaps not. Time will tell.

Either way, this isn't the place to go into this in depth, but it's something which I believe you'll see more and more debate about in the months and years to come. I personally plan to learn the lessons of Agile and try to apply them to my other kinds of engagements from now on. So what are the big lessons I've learned over the last 14 weeks?

Lessons Learned

1. Remember the “10 Commandments” [link]
2. While this project benefited from a number of favorable conditions (small and discrete and defined scope) I firmly believe that Agile methodologies can cope with far larger projects and less well defined scope and requirements. They are designed to manage the risks this brings while embracing the increased flexibility.
3. The quality was evident – we never had a major bug appear during demos to users. this instilled confidence in the Ambassador Users and it also helped the process in that the users could "play" with the system and understand the evolving requirements.
4. Have a demo box permanently available. This was requested a lot by our AU's but it was too difficult to achieve. But would have helped to get more users have hands on earlier on in the process, and help with wider user group buy in / adoption.
5. Don't however waste time setting up the demo box again and again – automate it (with Cruise Control and Maven)
6. One important aspect of an iterative process is that it can be continually reviewed throughout the project lifecycle. The team took advantage of this, resulting in an improved format for the workshops and new implementing reporting and testing procedures.
7. You must be disciplined

· Think about what you are doing and ask “Does it add value? Does it help towards delivery?”

· We didn't always follow our own standards. Sometimes it might have been better to enforce things with Subversion, Maven, PMD and Checkstyle.

8. It is essential that different user groups respect and understand the prioritisation objectives of others. This was certainly the case on this project.
9. It was also important that the Ambassador Users had the appropriate skills and experience to drive the project requirements forward and they were empowered to make decisions pertaining to the prioritisation and implementation of features.
10. It was of benefit to the AU's themselves to be round the same table though they come from different backgrounds and have different requirements. They gain a shared understanding through their participation

· What did the close developer / user bond require?:

(a) Developer communication skills – of highest importance – else you build lots, but you build the wrong thing

(b) Delivered code must be understandable to those who will interact with it

in the future

(c) Every feature built must be deployable - Even if the end result is not deployable (i.e. ours had no security so wouldn't go live)

(d) Buy in and understanding from all – Leads to a shared sense of responsibility

(e) Honesty from all – If you don't understand, say so; If you don't like, say so; ...

· What did the close developer / user bond result in?:

(f) It positively affected developers understanding of what will happen to their product once work is complete

(g) The developer sees who will use their system, rather than imagining them.

(h) The developer becomes aware of the change management, training and update issues after their work is complete

(i) Everyone is empowered

(j) Greater visibility for all participants

11. I, and others were amazed how much could be achieved in such a short period of time.
12. The lack of documentation in Agile projects makes people used to traditional approaches feel uncomfortable.
13. Communicate, communicate, communicate
14. You are not your code. Take pride in what you do, but let go once you check in.
15. Keep it simple. If it gets complicated, refactor.
16. Naming is very important. Agree on a strategy early on, then stick to it. If possible, enforce it with Checkstyle
17. Identify and stick to standards and best practices throughout
18. Tooling is important. If there's a plugin or a Wizard, you can save time.

· Frameworks save time, but add complexity. We didn't use one for MVC as things for us were simple.

19. Only use tools and frameworks as and when required

· But make sure you don't make them harder to adopt in the future if you decide to not use then from the get-go

Agile Development Ramblings: Part III
The Development

So far I've said a lot about the process, but nothing about the actual development. No prizes for guessing that this was different to my previous experiences too.

Development Environment Setup

We had to do this ourselves, so no time was wasted. From day one we had a server and by day two we were building both it and our development desktop systems with the software we required. We had few NFR's in this department apart from the requirement to use Websphere Portal and JSR168 Portlets. We also had to use Rational Application Developer 6.0 as our development IDE, and make sure our portlets were accessible and internationalisable. I do not think it would have been a problem had we had more NFR's, as long as the required media were made available to us. The fact we did not have to rely on anyone else; or an internal process was a boon.

We were left to our own devices. With access to the internet and an IBM Partner Web account we got to work. The systems we were given had nothing installed on them aside from Windows. We quickly decided to use Maven as the build and project management environment, MySQL and Hibernate for persistence, Subversion for configuration management, and CruiseControl for continuous integration. Each of the tools we used was picked after discussion between myself and the other developer. Installation tasks were split down the middle, with Notepad “what I did” notes being created and shared as we went.

This time didn't go entirely smoothly. RAD took ages to deploy and upgrade and we gave up getting the Portal Unit Test Environments (UTE's) to work on our desktops. (This turned out to be due to a known bug.) This however introduced me to the highly pragmatic approach to development any agile approach necessitates. You've got a problem; you have a think about what the relative costs and benefits of something are; if it's worth it, and doesn't endanger the delivery of features, go for it; if not, don't. If this is a real problem, you can flag it to the rest of the team and note it in the “Obstacles” section in the Sprint Plan. After such analysis, none of our problems were significant enough to reach the plan. In the case of the UTE's we decided that due to the small size of the team, we could use the shared server running a single instance of Portal and a single database for deployment and testing. It would require us to communicate constantly to make sure we didn't tread on each others toes, but this actually helped in 99% of situations. We never ended up developing in a vacuum.

All this meant that by the end of the first week we had an environment we were happy with, which we understood and would support our chosen approach.

The Coding

Again we took the agile approach. The aim was always to be developing and delivering as much as possible. At the outset, we set few constraints on ourselves. What we did have covered coding style, SVN commit protocol, Unit Testing coverage and approach and division of labour. Later on, as required, we bolted on Checkstyle and PMD, JWebUnit and JSTL. A unit testing protocol was forced on us as the default Maven target to produce our model jar file (“jar:deploy”) required that all unit tests passed. This proved invaluable and frequently caught problems in their infancy.

Architecture wise, we kept it light. Very light for someone like me who is a trained JEE architect. For the first two weeks, “MVC” was the only pattern that got mentioned. (We didn't use any MVC framework.) In the second iteration we introduced a Session Facade and in the fourth I got to use an Observer. We were lucky in that most of what we had to do was very simple. It was a no-brainer to split the code into two projects – one for the Model and another for the View / Controller. Again Maven made this simple. This Agile approach to architecture was a real revelation to me, and took some getting used to.

Build and Project management was light too. Maven was a boon in this respect. We only had to write thirteen lines of bespoke script in one project (to copy our Hibernate hbm.xml files from one place to another) and twenty in another (to call XMLAccess to deploy our portlet war to the Server). Everything else was already there for us, provided by the Maven plugins.

Maven also supported the project management, providing an automatically generated project website (via the “site” target) which allowed us to publish our how-tos to the wider team as we produced them. At the end of the development cycle, this proved an invaluable central repository for everything we produced over the proceeding thirteen weeks.

It is worth noting some of the practices I became familiar with during the development phase. The most fundamental was the aim to “write it once”, better still use a plugin or generate it. A good example follows:

Though we used MySQL as our data store, we never had to write any SQL. We didn't even have to write the hibernate mapping files. Instead, we generated them with XDoclet tags. To get from the tags to the schema, we called pre-existing Maven targets (“xdoclet:hibernatedoclet" and “hibernate:schema-export”) to generate our schema. This not only meant we got things going more quickly, and had fewer bugs (less code), but also we had automated what would otherwise have been very commonly executed task. Most importantly, these repetitive steps were taken out of our hands so there was less chance of losing valuable development time debugging what may turn out to be user error.

As with the setup phase I also had to focus 100% on delivery. I constantly asked myself “what is the value of what I am doing? Is it meeting the delivery goals? Is there something of more value I could be doing? Is there a better way to do what I am doing? Is it over redundant complicated / over engineered?”

All in all, we completed our six Sprints having delivered all the “Must Have” features and a large chunk of the “Should Have”s and “Could Have”s. We then moved into the final week for User Acceptance Testing and handover to the development partner. It was then that in my opinion our use of the Agile process showed some weaknesses.

The UAT

This however went very well. Every single AU completed all the tests, signing off every single point of functionality. As the process was agile, we were able to fix every minor bug which the testing highlighted before the final UAT, and also implement some minor “Could Have”s (such as clear buttons on search forms, default selections on drop downs, alphabetization of lists, etc.)

This was possible because we, as developers, had by this time formed a solid relationship with all our users. By seeing them really using the system in anger (we told them there were prizes for breaking it) we gained a further depth of understanding of the real usability of our system. This is something I personally have rarely experienced and it was enlightening. The project team agreed that if we had known the power of this earlier, we would have got the users to have one to one hands on sessions more frequently, earlier in the process. The benefits of this to the final product are hard to pin down quantitatively, but self evident in the many testimonials we received on the completed UAT forms.

Agile Development Ramblings: Part II
The Process
In this project I was an Architect / Developer. The rest of the development team comprised a developer / SCRUM “Master” (see below), a Project Manager, a Business Analyst, a Data Architect, and an Agile Process Consultant. As with all projects, we reported into a Programme Manager. This was the first agile project for this customer. Consequently, we were proving the process as well as the product.

The Agile methodology we used was a mixture of SCRUM and DSDM. DSDM is an iterative methodology based on the core concepts of time boxed, prioritised development.

Elements of SCRUM were used to augment the internal iteration process and structure our day to day existence: Progress was followed using a “Burn Down Chart”, “Product -” and “Sprint - Backlogs” were created, (although these used the DSDM MoSCoW methodology) – See below for more detail on all these elements.

Notes on SCRUM
All the work to be done on a SCRUM project is recorded in the “Product Backlog”. This is a list of all the desired features for the final product. At the end of each Sprint a “Sprint Review Meeting” (see below) is held during which the Ambassador Users – in our case representatives of each of the organisations which will use our system - prioritise the remaining items on this Backlog. (see below for more detail).

This prioritised feature list is then taken by the Development Team who select the features they will target to complete in the next Sprint. These features are then moved from the Product Backlog to the “Sprint Backlog”.

2 Week Sprints
In SCRUM, iterations are called “Sprints”. Ours were 2 weeks long. (The length can be varied but it is recommended that they stay the same length throughout the project.) This worked well for us in a number of ways: they were regular, so the AU's could predict when they had to travel to London, they were long enough that a real chunk of functionality could be developed and then demoed, but short enough that everything done in development had to be relevant. They meant we stayed very focused throughout the 14 weeks.

The Sprint Plan, Estimation, and the Burn Down Chart
During the Sprint the team stays on track by holding brief daily meetings. The first meeting after the Sprint Planning Meeting is slightly different. Here, the team works down the Ambassador User prioritised feature list, breaking each feature down into manageable and unitary tasks. Each of these tasks is then entered into the Sprint Plan “Sprint Backlog” sheet as an individual item (see below). In addition, meetings, demos and any other requirements on team member’s time are added. It is important that all time spent during the project is tracked on the Sprint Plan.

The Sprint plan is a very simple, yet highly effective thing. It's an excel spreadsheet. The “Sprint Backlog” page is where all the action happens. Firstly the start and end of the Sprint are entered. This is used to calculate the time available. Each member of the development team (including project managers and business analysts) then enters their details in the top right hand corner. These details include their initials (used later in task allocation), their predicted availability for the project (e.g. a full-time developer is usually 90%), and any unavailable hours (i.e. holidays) are recorded as a negative number and deducted from the total available time. This information is then used to calculate the time each member has available to contribute to the project that Sprint (the “Avail” row).

The first allocation and estimation of each task then takes place. The team as a whole works through each of the entered task items in turn. For each, an “Owner” is allocated. They then estimate the time in hours they feel it will take them to complete the task.

The final step before the initial Sprint Planning is complete is to check that no team member is allocated more hours of work than they are capable of completing. This can be seen by comparing the “Avail” and “Alloc” rows for each member. If there is am imbalance in allocations, tasks can be re allocated. If reallocation is impossible then task items for the overworked member can be taken out of scope.

Finally, the effect of all this information can be seen simply and graphically by looking at the Burn Down Chart (“Sprint Chart”) which plots total estimated work remaining versus the time left in the Sprint. In a glance this lets everyone see the current estimated time versus the available time. Both these variables has a line. If your “estimated time” line is below the “available time” line then you're very happy. Too far below, and more tasks (and therefore features) can be brought into scope so everyone is well utilised. In the nightmare scenario where you're estimating above the “available time” line you know you (estimate) you have more work than there is time to do it. In this case you generally give things a few days to see if you can catch up with the line. If this fails, the team needs to discuss and take task items out of scope.

(Re)Estimation
Due to the simplicity and speed of the estimation process once the information is first entered, it is then possible to repeat this task every day – first thing in the morning is best. Each task that is not completed or is in scope has it's time to complete re-estimated. If a task has been completed since the last meeting, the estimate is set at “0”. As before, individual workloads can be checked and reallocation and de / re scoping used to ensure all workloads are manageable. As with the first session, the meeting ends with a look at the Sprint Chart and it's time lines.

This construction and updating of the Sprint Plan, especially the viewing of “the line” had the very pronounced effect of fostering a shared sense of responsibility in the team for everything everyone is doing. It was also very visible. Discussions which might have before happened in hidden Project Manager cabals are out in the open, and can be discussed by all. The whole team could not avoid the fact that things need to react to the current state of progress. This all resulted in a group who were all very focused on delivery. Not only that, but everyone was aware of what has to be delivered and what the workloads were like to reach that point. Everyone was responsible for the Sprint's success.

Is the fact that everyone has to attend a meeting a problem in itself? I'd say no. Things were always quick and relevant. Again the Sprint Plan ensured that. They were also quick. Fifteen minutes was usually all it took. Any time longer was when something of great importance to all has to be discussed.

Sprint Review Meetings
At the end of each Sprint the team demonstrated to the assembled AU’s the completed functionality of the real system, and outstanding and new features were discussed and prioritised (see below). We did this by showing the current system in all it's glory. It was my fear going into the first of these meetings that there would be nothing to show. I was wrong. What little we had done was ecstatically received. We had started to build what had been asked for, no matter how little we had actually achieved.

What's more, in the process of doing this we (the developers) had to explain our system. We asked questions to clarify our understanding in areas (which were many to begin with) of the business domain. We even got a user up to show us how they would do things on our fledgling system in their job.

It is worth noting that when I and the other developer joined the project, things had been running for a while. The Ambassador Users had been enlisted months before and had been along to a few meetings where the process had been explained to them and they had discussed and initially prioritised the planned high level features. Feedback gathered at the end showed this had left them confused and de motivated. Even when we did join the project, we did not to meet them for three weeks. These were spent setting up our development environments (1 week) and developing the first iteration (2 weeks). It transpired that show and tell was by far the best way to engage them and keep them interested.

However, over time, all the project team developed a relationship and shared sense of responsibility with the AU's. They felt engaged, and we gained an understanding of who they were and how they wanted to use the new system. This was particularly useful in our situation as we had such a diverse user base, drawn from seven different organisations, each of whom needed to use the system in a different way. In addition, we couldn't obfuscate and hide behind excuses and jargon because they were sitting in front of us. Similarly they had to think about what they wanted and if we gave them something that they asked for and it turned out that they had explained it badly, they realised the benefit of carefully thinking things through when making requests.

Feature Prioritisation with MoSCoW
“MoSCoW” is a way of prioritizing features. In the context of DSDM the MoSCoW technique is used to prioritize requirements. It is an acronym that stands for:

• MUST have this requirement to meet the business needs.


• SHOULD have this requirement if at all possible, but the project success does not reply on this.


• COULD have this requirement if it does not affect the fitness of business needs of the project.


• WOULD have this requirement at later date if there is some time left (or in the future development of the system).

These criteria were used by the Ambassador Users to prioritise the outstanding elements on the Product Backlog at the end of each Sprint Planning Session. This then informed the features which we the developers picked off for development in each Sprint.

The approach benefits from being simple. Everyone could use it and more importantly could remember the terminology. They also were aware of the ramifications of allocating the different levels of priority to their requirements. Not everything asked for was developed.

Agile Development Ramblings: Part I
Introduction
I've just completed my first Agile development project. We developed a fully functional prototype Java Portlet app in 14 weeks. We finished on budget, ahead of schedule and delivered all the initial requirements plus extras.

In addition, our Ambassador Users gave us the following feedback:

• “It is difficult to say how we could improve the process”.


• “I’ve been away for a few weeks and so much has happened”


• "The meetings I attended have been interesting and informative. The IT people seem to be bending over backwards to try and supply us with our needs and wants.”


• “I feel quite excited about how useful the portal will be when the programme is implemented. I know it's difficult to say but I really do think from what I've seen that Probation staff's job will be much assisted by the new system and feel that the time saved by using the portal is going to be phenomenal”


• "I have found the whole User Ambassador process to be an extremely interesting, enjoyable and enlightening experience."


• "I've recommended to my Organisation that we work this way in the future."


• “Our input is being used in shaping the Portal through the content and its usability resulting in a system in which all users will see the end product as a valuable resource”


• "I feel a great sense of achievement in what we have produced in such a short period of time.”

This is the first development project I've worked on which has been anywhere near as successful. I would completely attribute this to the Agile approach which was used. At the outset, my boss asked me to keep a note of how things went so I could share it with the team. This is the result.