Recent Developments in Nonregular Fractional Factorial Designs

Nonregular fractional factorial designs such as Plackett-Burman designs and other orthogonal arrays are widely used in various screening experiments for their run size economy and flexibility. The traditional analysis focuses on main effects only. Hamada and Wu (1992) went beyond the traditional approach and proposed an analysis strategy to demonstrate that some interactions could be entertained and estimated beyond a few significant main effects. Their groundbreaking work stimulated much of the recent developments in design criterion creation, construction and analysis of nonregular designs. This paper reviews important developments in optimality criteria and comparison, including projection properties, generalized resolution, various generalized minimum aberration criteria, optimality results, construction methods and analysis strategies for nonregular designs.


Introduction
In many scientific investigations, the main interest is in the study of effects of many factors simultaneously. Factorial designs, especially two-level or three-level factorial designs, are the most commonly used experimental plans for this type of investigation. A full factorial experiment allows all factorial effects to be estimated independently. However, it is often too costly to perform a full factorial experiment, so a fractional factorial design, which is a subset or fraction of a full factorial design, is preferred since it is cost-effective.
Fractional factorial designs are classified into two broad types: regular designs and nonregular designs. Regular designs are constructed through defining relations among factors and are described in many textbooks such as Box, Hunter and Hunter (2005), Dean and Voss (1999), Montgomery (2005) and Wu and Hamada (2000). These designs have a simple aliasing structure in that any two effects are either orthogonal or fully aliased. The run sizes are always a power of 2, 3 or a prime, and thus the "gaps" between possible run sizes are getting wider as the power increases. The concept of resolution (Box and Hunter 1961) and its refinement minimum aberration (Fries and Hunter 1980) play a pivotal role in the optimal choice of regular designs. There are many recent developments on minimum aberration designs; see Wu and Hamada (2000) and Mukerjee and Wu (2006) for further references.
Nonregular designs such as Plackett-Burman designs and other orthogonal arrays are widely used in various screening experiments for their run size economy and flexibility (Wu and Hamada, 2000). They fill the gaps between regular designs in terms of various run sizes and are flexible in accommodating various combinations of factors with different numbers of levels. Unlike regular 1 designs, nonregular designs may exhibit a complex aliasing structure, that is, a large number of effects may be neither orthogonal nor fully aliased, which makes it difficult to interpret their significance. For this reason, nonregular designs were traditionally used to estimate factor main effects only but not their interactions. However, in many practical situations it is often questionable whether the interaction effects are negligible. Hamada and Wu (1992) went beyond the traditional approach and proposed an analysis strategy to demonstrate that some interactions could be entertained and estimated through their complex aliasing structure. They pointed out that ignoring interactions can lead to (i) important effects being missed, (ii) spurious effects being detected, and (iii) estimated effects having reversed signs resulting in incorrectly recommended factor levels.
Much of the recent studies in nonregular designs were motivated from results in Hamada and Wu (1992). They included proposal of new optimality criteria, construction and analysis of nonregular designs. The primary aim of this paper is to review major developments in nonregular fractional factorial designs since 1992.
Here is a brief history of the major developments in nonregular designs. Plackett and Burman (1946) gave a large collection of two-level and three-level designs for multi-factorial experiments. These designs are often referred to as the Plackett-Burman designs in the literature. Rao (1947) introduced the concept of orthogonal arrays, including Plackett-Burman designs as special cases. Cheng (1980) showed that orthogonal arrays are universally optimal for main effects model. Hamada and Wu (1992) successfully demonstrated that some interactions could be identified beyond a few significant main effects for Plackett-Burman designs and other orthogonal arrays. Lin and Draper (1992) studied the geometrical projection properties of Plackett-Burman designs while Wang and Wu (1995) and Cheng (1995Cheng ( , 1998) studied the hidden projection properties of Plackett-Burman designs and other orthogonal arrays. The hidden projection properties provide an explanation for the success of the analysis strategy due to Hamada and Wu (1992). Sun and Wu (1993) were the first to coin the term "nonregular designs" when studying statistical properties of Hadamard matrices of order 16. Deng and Tang (1999) and Tang and Deng (1999) introduced the concepts of generalized resolution and generalized minimum aberration for two-level nonregular designs. Xu and Wu (2001) proposed the generalized minimum aberration for mixed-level nonregular designs. Because of the popularity of minimum aberration, the research on nonregular designs has been largely focused on the construction and properties of generalized minimum aberration designs. Our reference list suggests that keen interest in nonregular designs began in 1999 and continues to this day as evident by the increasing number of scientific papers on nonregular designs in major statistical journals. Section 2 reviews the data analysis strategies for nonregular designs. Section 3 discusses the geometrical and hidden projection properties of the Plackett-Burman designs and other orthogonal arrays. Section 4 introduces the generalized resolution and generalized minimum aberration and their statistical justifications. Section 5 introduces the minimum moment aberration criterion, another popular criterion for nonregular designs. Section 6 considers uniformity and connections with various optimality criteria. Section 7 reviews construction methods and optimality results. Section 8 gives concluding remarks and future directions. 2

Analysis Strategies
We begin with a review of a breakthrough approach (Hamada and Wu 1992) by entertaining interactions in Plackett-Burman designs and other orthogonal arrays after identifying a few important main effects. Then we review another strategy proposed by Cheng and Wu (2001) for the dual purposes of factor screening and response surface exploration (or interaction detection) with quantitative factors.
The analysis strategy proposed by Hamada and Wu (1992) consists of three steps.
Step 1. Entertain all the main effects and interactions that are orthogonal to the main effects. Use standard analysis methods such as ANOVA and half-normal plots to select significant effects.
Step 2. Entertain the significant effects identified in the previous step and the two-factor interactions that consist of at least one significant effect. Identify significant effects using a forward selection regression procedure.
Step 3. Entertain the significant effects identified in the previous step and all the main effects. Identify significant effects using a forward selection regression procedure.
Iterate between Steps 2 and 3 until the selected model stops changing. Note that the traditional analysis of Plackett-Burman or other nonregular designs ends at Step 1. Hamada and Wu (1992) based their analysis strategy on two empirical principles, effect sparsity and effect heredity (see Wu and Hamada 2000, Section 3.5). Effect sparsity implies that only few main effects and even fewer two-factor interactions are relatively important in a factorial experiment. Effect heredity means that in order for an interaction to be significant, at least one of its parent factors should be significant. Effect heredity excludes models that contain an interaction but none of its parent main effects, which lessens the problem of obtaining uninterpretable models. Hamada and Wu (1992) wrote that the strategy works well when both principles hold and the correlations between partially aliased effects are small to moderate. The effect sparsity suggests that only a few iterations will be required.
Using this procedure, Hamada and Wu (1992) reanalyzed data from three real experiments, a cast fatigue experiment using a 12-run Plackett-Burman design with seven 2-level factors, a blood glucose experiment using an 18-run mixed-level orthogonal array with one 2-level and seven 3-level factors, and a heat exchange experiment using a 12-run Plackett-Burman design with ten 2-level factors. They demonstrated that the traditional main effects analysis was limited and the results were misleading.
For illustration, consider the cast fatigue experiment conducted by Hunter et al. (1982) that used a 12-run Plackett-Burman design to study the effects of seven factors (A-G) on the fatigue life of weld repaired castings. Table 1 gives the data matrix and responses, where columns 8-11 are not used. The original analysis by Hunter et al. (1982) identified two significant factors F and D. The factor D had a much smaller effect with a p value around 0.2. The fitted model waŝ with a R 2 = 0.59. However, Hunter et al. (1982) noted a discrepancy between their fitted model (1) and previous work, namely, the sign of factor D was reversed. Applying the three-step analysis strategy, Hamada and Wu (1992) identified a significant two-factor interaction F G and obtained the following modelŷ This model has R 2 = 0.89, which is a significant improvement over model (1)  For quantitative factors with more than two levels, Cheng and Wu (2001) proposed the following two-stage analysis strategy to achieve the dual objectives of factor screening and response surface exploration (or interaction detection) using a single design. This two-stage analysis strategy is also the two key aspects in standard response surface methodology. For m quantitative factors, denoted by x 1 , . . ., x m , the second-order model is where β 0 , β i , β ii , β ij are unknown parameters and is the error term.  Cheng and  proposed that the two-stage analysis be broken down into three parts: screening analysis in stage 1, projection that links stages 1 and 2, and response surface exploration in stage 2. Various screening analyses can be utilized in stage 1, such as the conventional ANOVA or half-normal plots on the main effects. Their analysis strategy again assumes that effect sparsity and effect heredity principles hold. They reanalyzed a PVC insulation experiment reported by Taguchi (1987) that used a regular 27-run design with nine 3-level factors. They identified a significant linear-by-linear interaction effect which was missed by Taguchi.
For illustration, consider an experiment reported by King and Allen (1987) that used an 18-run orthogonal array to study the effects of one two-level factor (A) and seven three-level factors (B-H) on radio frequency chokes. Each run had two replicates and Table 2 gives the design matrix and the responses. Xu, Cheng and Wu (2004) performed data analysis following the two-stage analysis strategy. At the first stage, they fitted an ANOVA model for main effects and found that four factors B, E, G, and H were significant at the usual 5% level. At the second stage, they fitted a second-order model among the four active factors and obtained the following nine-effect model: where the levels 0, 1, 2 were coded as −1, 0, 1, respectively. The model has R 2 = 0.96, indicating a good fit. It is worthwhile to point out that the 18-run design does not have enough degrees of freedom to estimate all six two-factor interactions among four factors (since each two-factor interaction has four degrees of freedom).
More sophisticated analysis strategies have been proposed for experiments with complex aliasing. Chipman et al. (1997) proposed a Bayesian approach that employs a Gibbs sampler to perform an efficient stochastic search of the model space. Many other recent variable selection methods can also be used for analyzing nonregular designs. For example, Yuan et al. (2007) suggested an extension of the general-purpose LARS (least angle regression), first proposed by Efron et al. (2004).

Projection Properties of Plackett-Burman Designs and Other Orthogonal Arrays
We first review the concept of orthogonal arrays due to Rao (1947). An orthogonal array of N runs, m factors, s levels and strength t, denoted by OA(N, s m , t), is an N × m matrix in which each column has s symbols or levels and for any t columns all possible s t combinations of symbols appear equally often in the matrix. Rao (1973) generalized the definition to the asymmetrical case where an orthogonal array is allowed to have variable numbers of symbols, i.e., mixed levels. For example, the 12-run Plackett-Burman design in Table 1 is an OA(12, 2 11 , 2) and the 18-run design in Table  2 is an OA(18, 2 1 3 7 , 2). Hedayat, Sloane and Stufken (1999) gave a comprehensive description on various aspects of orthogonal arrays. Plackett-Burman designs are saturated orthogonal arrays of strength two because all degrees of freedom are utilized to estimate main effects. Orthogonal arrays of strength two allow all the main effects to be estimated independently and they are universally optimal for the main effects model (Cheng 1980). A necessary condition for the existence of an OA(N, In the literature, orthogonal arrays of strength two are often called orthogonal designs or orthogonal arrays without mentioning the strength explicitly.
Orthogonal arrays include both regular and nonregular designs. For regular designs, the concepts of strength and resolution are equivalent because a regular design of resolution R is an orthogonal array of strength t = R − 1. For a regular design of resolution R, the projection onto any R factors must be either a full factorial or copies of a half-replicate of a full factorial. The projection for nonregular designs is more complicated.
Plackett-Burman designs are of strength two so that the projection onto any two factors is a full factorial. Lin and Draper (1992) studied the geometrical projection properties of the Plackett-Burman designs onto three or more factors. Their computer searches found all the projections of 12-, 16-, 20-, 24-, 28-, 32-and 36-run Plackett-Burman designs onto three factors. They found that these projections must have at least a copy of the full 2 3 factorial or at least a copy of a 2 3−1 replicate or both. In particular, any projection onto three factors must contain a copy of a full factorial except for the 16-and 32-run Plackett-Burman designs, which are regular designs. The important statistical implication of this finding is that if only at most three factors are truly important, then after identifying the active factors, all factorial effects among these active factors are estimable, regardless which three factors are important. Box and Tyssedal (1996) defined a design to be of projectivity p if the projection onto every subset of p factors contains a full factorial design, possibly with some points replicated. It follows from these definitions that an orthogonal array of strength t is of projectivity t. Cheng (1995) showed that, as long as the run size N is not a multiple of 2 t+1 , an OA(N, 2 m , t) with m ≥ t + 2 has projectivity t + 1, even though the strength is only t.
The 12-run Plackett-Burman design given in Table 1 is of projectivity three but not of projectivity four. Wang and Wu (1995) found that its projection onto any four factors has the property that all the main effects and two-factor interactions can be estimated if the higher-order interactions are negligible. They referred this estimability of interactions without relying on geometric projection to as having a hidden projection property.
More generally, Wang and Wu (1995) defined a design as having a hidden projection property if it allows some or all interactions to be estimated even when the projected design does not have the right resolution or other geometrical/combinatorial design property for the same interactions to be estimated. For the Plackett-Burman designs their hidden projection property is a result of complex aliasing between the interactions and the main effects. For example, in the 12-run Plackett-Burman design given in Table 1, any two-factor interaction, say AB, is orthogonal to the main effects A and B, and partially aliased with all other main effects with correlation 1/3 or −1/3. Because no two-factor interaction is fully aliased with any main effects, it is possible to estimate four main effects and all six two-factor interactions among them together.
The general results on hidden projection properties were obtained by Cheng (1995Cheng ( , 1998) and Bulutoglu and Cheng (2003). Cheng (1995) showed that as long as the run size N of an OA(N, 2 m , 2) is not a multiple of 8, its projection onto any four factors allows the estimation of all the main effects and two-factor interactions when the higher-order interactions are negligible. Bulutoglu and Cheng (2003) showed that the same hidden projection property also holds for Paley designs [constructed by a method due to Paley (1933)] of sizes greater than 8, even when their run sizes are multiples of 8. A key result is that such designs do not have defining words of length three or four. Cheng (1998) further showed that as long as the run size N of an OA(N, 2 m , 3) is not a multiple of 16, its projection onto any five factors allows the estimation of all the main effects and two-factor interactions. Cheng (2006) gave a nice review of projection properties of factorial designs and their role in factor screening.
A few papers studied projection properties of designs with more than two levels. Wang and Wu (1995) studied the hidden projections onto 3 and 4 factors of the popular OA(18, 3 7 , 2) given in Table 2 (columns B-H). Cheng and Wu (2001) further studied the projection properties of this OA(18, 3 7 , 2) and an OA(36, 3 12 , 2) in terms of their two-stage analysis strategy. They constructed a nonregular OA (27, 3 8 , 2) that allows the second-order model to be estimated in all four-factor projections. In contrast, any regular 27-run design with eight 3-level factors does not have this four-factor projection property. They concluded that three-level nonregular designs have better projection properties and are more useful than regular designs for the dual purposes of factor screening and response surface exploration. Xu, Cheng and Wu (2004) further explored the projection properties of 18-run and 27-run orthogonal arrays and constructed a nonregular OA (27, 3 13 , 2) that allows the second-order model to be estimated in all of the five-factor projections. Tsai et al. (2000Tsai et al. ( , 2004 and Evangelaras et al. (2005Evangelaras et al. ( , 2007 also studied projection properties of three-level orthogonal arrays. Dey (2005) studied projectivity properties of asymmetrical orthogonal arrays with all except one factors having two levels.

Generalized Resolution and Generalized Minimum Aberration
Prior to 1999, an outstanding problem was how to assess, compare and rank nonregular designs in a systematic fashion. Deng and Tang (1999) and Tang and Deng (1999) were the first to propose generalized resolution and generalized minimum aberration criteria for 2-level nonregular designs, which are natural generalizations of the traditional concepts of resolution and minimum aberration for regular designs.
To define these two important concepts, generalized resolution and generalized minimum aber-ration, Deng and Tang (1999) and Tang and Deng (1999) introduced the important notion of J-characteristics and is defined as follows. Given a two-level N × m design D = (d ij ), for where c ij is the ith component of column c j . The quantity j k (s)/N can be viewed as an extension of correlation. For illustration, consider the 12-run Plackett-Burman design given in Table 1.  (2004) showed that the reserve is also true. Therefore, for a nonregular design, there always exist some s such that 0 < ρ k (s) < 1. Suppose that r is the smallest integer such that max |s|=r J r (s) > 0, where the maximization is over all subsets of r columns. Then the generalized resolution is defined to be For the 12-run design in Table 1, r = 3, δ = 2/3 and the generalized resolution is R = 3.67. It is easy to see that for an OA(N, 2 m , t), j k (s) = 0 for any k ≤ t and therefore r ≤ R < r + 1 where r = t + 1. If δ > 0, a subset s of D with r columns contains at least N δ/2 r copies of a full 2 r factorial and therefore the projectivity of D is at least r (Deng and Tang 1999). For a regular design, δ = 0 and the projectivity is exactly r − 1. Two regular designs of the same resolution can be distinguished using the minimum aberration criterion, and the same idea can be applied to nonregular designs using the minimum G-aberration criterion . Roughly speaking, the minimum G-aberration criterion always chooses a design with the smallest confounding frequency among designs with maximum generalized resolution. Formally, the minimum G-aberration criterion is to sequentially minimize the components in the confounding frequency vector where f kj denotes the frequency of k-column combinations s with J k (s) = N + 1 − j.
Minimum G-aberration is very stringent and it attempts to control J-characteristics in a very strict manner. Tang and Deng (1999) proposed a relaxed version of minimum G-aberration and called it the minimum G 2 -aberration criterion. Let The vector (A 1 (D), . . . , A m (D)) is called the generalized wordlength pattern, because for a regular design D, A k (D) is the number of words of length k in the defining contrast subgroup of D. The minimum G 2 -aberration criterion  is to sequentially minimize the generalized wordlength pattern A 1 (D), A 2 (D), . . . , A m (D). For regular designs both minimum G-aberration and minimum G 2 -aberration criteria reduce to the traditional minimum aberration criterion. However, these two criteria can result in selecting different nonregular designs. We note that minimum G-aberration nonregular designs always have maximum generalized resolution whereas minimum G 2 -aberration nonregular designs may not. This is in contrast to regular case where minimum aberration regular designs always have maximum resolution among all regular designs. Tang and Deng (1999) also defined minimum G e -aberration for any e > 0 by replacing J 2 k (s) with J e k (s) in (5). However, only the minimum G 2 -aberration criterion is popular due to various statistical justifications and theoretical results. Xu and Wu (2001) proposed the generalized minimum aberration criterion for comparing asymmetrical (or mixed-level) designs. The generalized minimum aberration criterion was motivated from ANOVA models and includes the minimum G 2 -aberration criterion as a special case. By exploring an important connection between design theory and coding theory, Xu and Wu (2001) showed that the generalized wordlength pattern defined in (5) are linear combinations of the distribution of pairwise distance between the rows. This observation plays a pivotal role in the subsequent theoretical development of nonregular designs. Ma and Fang (2001) independently extended the minimum G 2 -aberration criterion for designs with more than two levels. They named their criterion as the minimum generalized aberration criterion, which is a special case of the generalized minimum aberration criterion proposed by Xu and Wu (2001). Ye (2003) redefined the generalized wordlength pattern and generalized minimum aberration for two-level designs using indicator functions.  defined generalized resolution and generalized minimum aberration criterion for quantitative factors. The generalized minimum aberration criterion proposed by Xu and Wu (2001) is independent of the choice of treatment contrasts and thus model-free whereas the generalized minimum aberration criterion by  depends on the specific model.

Statistical Justifications
Deng and Tang (1999) provided a statistical justification for the generalized resolution by showing that designs with maximum generalized resolution minimize the contamination of nonnegligible two-factor interactions on the estimation of main effects. Tang and Deng (1999) provided a similar statistical justification for minimum G 2 -aberration designs. In a further extension, Xu and Wu (2001) gave a statistical justification for generalized minimum aberration designs with mixed levels.
A common situation that arises in practice is that the main effects are of primary interest but there are uninteresting yet non-negligible interactions that we know will affect the main effects estimates. To fix ideas, consider a two-level N × m design D = (d ij ) with columns denoted by d 1 , . . . , d m and generalized resolution between 3 and 4. Suppose that one fits a main effects model but the true model is 9 The least squares estimatorβ j of β j from the working model (6), under the true model (7), has expectation given by (3). There are many ways to minimize the biases in estimating main effects due to the presence of the interaction effects. A conservative approach is minimizing the maximum bias, max j<k<l J 3 (d j , d k , d l ). This is equivalent to maximizing the generalized resolution as defined in (4). Therefore, designs with maximum generalized resolution minimize the maximum bias of nonnegligible interactions on the estimation of the main effects. A more aggressive approach is minimizing the sum of squared coefficients

Minimum Moment Aberration
Based on coding theory, Xu (2003) proposed the minimum moment aberration criterion for assessing nonregular designs. For an N × m design D with s levels and a positive integer t, define the tth power moment to be where δ ij (D) is the number of coincidences between the ith and jth rows. For two row vectors (x 1 , . . . , x m ) and (y 1 , . . . , y m ), the number of coincidences is the number of i's such that x i = y i . Note that m − δ ij (D) is known as the Hamming distance between the ith and jth rows in coding theory. The power moments measure the similarity among runs (i.e., rows). The first and second power moments measure the average and variance of the similarity among runs. Minimizing the power moments makes runs to be as dissimilar as possible. Therefore, good designs should have small power moments. This leads to the minimum moment aberration criterion (Xu, 2003) that is to sequentially minimize the power moments K 1 (D), K 2 (D), . . . , K m (D).
We note that the computation of the power moments involves the number of coincidences between rows. By applying generalized MacWilliams identities and Pless power moment identities, two fundamental results in coding theory (see, e.g., MacWilliams and Sloane 1977, chap. 5), Xu (2003) showed that the power moments K t defined in (8) are linear combinations of the wordlength patterns A 1 , . . . , A t in (5). Specifically, where c i are constants depending on i, N, m, s only and the leading coefficient c t is positive. It is not difficult to see now that sequentially minimizing K 1 (D), . . . , K m (D) is equivalent to sequentially minimizing A 1 (D), . . . , A m (D). Therefore, the minimum moment aberration is equivalent to the generalized minimum aberration.
The equivalence of the minimum moment aberration and the generalized minimum aberration is very important. On the one hand, it not only provides a geometrical justification for the generalized minimum aberration, but also provides a statistical justification for the minimum moment aberration. On the other hand, it provides a useful tool for efficient computation and theoretical development. For an N × m design with two levels, the complexity of computing the generalized wordlength pattern according to the definition (5) is O(N 2 m ) whereas the complexity of computing m power moments is O(N 2 m 2 ). The saving in computation is tremendous when the number of factors m is large. This observation led to successful algorithmic constructions of mixed-level orthogonal arrays (Xu 2002), a catalog of 3-level regular designs (Xu 2005b), and blocked regular designs with minimum aberration (Xu and Lau 2006). As a theoretical tool, Xu (2003) developed a unified theory for nonregular and supersaturated designs. Xu and Lau (2006) and Xu (2006) further used the concept of minimum moment aberration to develop a theory for blocked regular designs and constructed minimum aberration blocked regular designs with 32, 64 and 81 runs.
To mimic the minimum G-aberration criterion (Deng and Tang 1999), Xu and Deng (2005) applied the minimum moment aberration criterion to projection designs and proposed the moment aberration projection to rank and classify general nonregular designs. It was a surprise that the minimum G-aberration criterion and the moment aberration projection criterion are not equivalent for two-level designs. Xu and Deng (2005) provided examples to show that the latter is more powerful in classifying and ranking nonregular designs than the former. They also provided examples to illustrate that the moment aberration projection criterion is supported by other design criteria. The concept of moment projection turns out to be very useful in the algorithmic construction of regular designs; see Xu (2005bXu ( , 2007. For mixed-level designs, Xu (2003) suggested to weight each column according to its level, called natural weights, and replace δ ij (D) in (8) with the number of weighted coincidences. Xu (2003) showed that the minimum moment aberration with natural weights is weakly equivalent to the generalized minimum aberration for mixed-level designs.

Uniformity and Connection Among Various Criteria
Uniformity or space filling is a desirable design property for computer experiments (Fang, Li and Sudjianto 2006). Various uniformity measures are used to assess the space filling property for the so-called uniform design (Fang and Wang 1994;. Fang and Mukerjee (2000) found a connection between aberration and uniformity for 2-level regular designs. This connection was extended by Ma and Fang (2001) for general two-level designs. The basic result states that for a two-level N × m design D, regular or nonregular, the centered L 2 -discrepancy (CL 2 ), a uniformity measure introduced by Hickernell (1998), can be expressed in terms of its generalized wordlength pattern A k (D) as follows: Since the coefficient of A k (D) decreases exponentially with k, one can anticipate that designs with small A k (D) for small values of k should have small {CL 2 (D)} 2 ; in other words, minimum G 2 -aberration designs tend to be uniform over the design region. Ma and Fang (2001) also gave analytic formulas that link the generalized wordlength pattern with other uniformity measures for two-and three-level designs. Tang (2001) showed that minimum G 2 -aberration designs have good low-dimensional projection properties. Ai and Zhang (2004a) extended this result to mixed-level designs and showed that generalized minimum aberration designs have good low-dimensional projection properties.
There is much more work on the connection among aberration, uniformity and projection. Hickernell and Liu (2002) showed that generalized minimum aberration designs and minimum discrepancy designs are equivalent in a certain limit. Qin and Fang (2004)

Construction and Optimality Results
An important and challenging issue is the construction of good nonregular designs. There are two simple reasons: (i) nonregular designs do not have a unified mathematical description; (ii) there are much more nonregular designs than regular designs. Since 1999, a main stream of researches focused on searching or constructing nonregular designs with good properties in terms of the minimum G 2 -aberration and generalized minimum aberration criteria. This section reviews algorithmic constructions and optimality results. The last subsection reviews a simple yet powerful construction method via quaternary codes.

Algorithmic constructions
Two-level nonregular designs are often constructed from Hadamard matrices. A Hadamard matrix of order N is an N × N matrix with the elements ±1 whose columns (and rows) are orthogonal to each other. From a Hadamard matrix of order N , one obtains a saturated two-level orthogonal array with N runs and N − 1 columns, which is a nonregular design if N is not a power of 2. Neil Sloane of AT&T Shannon Labs maintains a large collection of Hadamard matrices at his website http://www.research.att.com/∼njas/, which includes all Hadamard matrices of orders N up through 28, and at least one of every order N up through 256. Sloane also maintains a library of orthogonal arrays as a companion to the book by Hedayat, Sloane and Stufken (1999). SAS maintains a library of orthogonal arrays (of strength two) up through 144 runs at http://support.sas.com/techsup/technote/ts723.html. SAS also provides a set of free macros for constructing over 117,000 orthogonal arrays up through 513 runs, which are documented in the free Web book by Kuhfeld (2005).
A simple strategy for constructing generalized minimum aberration designs is searching over all possible projection designs from existing Hadamard matrices or orthogonal arrays. Deng and Tang (2002) presented a catalog of generalized minimum aberration designs by searching over Hadamard matrices of order 16, 20, and 24. However, limiting to Hadamard matrices may miss the optimal design in some cases; therefore, Li, Tang and Deng (2004) searched generalized minimum aberration designs from and outside Hadamard matrices with 20, 24, 28, 32 and 36 runs. They found that the best 20 × 6 and 20 × 7 designs according to minimum G-aberration cannot be obtained from Hadamard matrices. Similarly, Xu and Deng (2005) considered the construction of optimal designs under the moment aberration projection criterion. Besides searching over all Hadamard matrices of order 16 and 20, they searched over all projection designs from 68 saturated OA(27, 3 13 , 2) from Lam and Tonchev (1996). They also observed that not all 20-run and 27-run moment aberration projection designs can be embedded into Hadamard matrices or saturated orthogonal arrays.
Sun, Li and Ye (2002) proposed an algorithm for sequentially constructing non-isomorphic orthogonal designs. Two designs are said to be isomorphic or equivalent if one design can be obtained from the other by row permutations, column permutations, or relabeling of levels. An essential element of their algorithm is using minimal column base to reduce the computations for determining isomorphism between any two designs. By using this algorithm, they obtained the complete catalogs of two-level orthogonal designs for 12, 16, and 20 runs. Their results suggest that there is only one unique 12 × m design for m = 4 and 7 ≤ m ≤ 11 and that there are two non-isomorphic 12 × m design for m = 5 and 6. All these designs can be found as projection designs of the 12-run Plackett-Burman design given in Table 1. They found that there are five 16 × 15 orthogonal designs, which are equivalent to the five non-isomorphic Hadamard matrices of order 16 by Hall (1961). An important result is that all 16-run orthogonal designs are projections of one of the five 16-run Hadamard matrices. They found that there are three 20 × 19 orthogonal designs, which are equivalent to the three non-isomorphic Hadamard matrices of order 20 by Hall (1965). From their complete catalog, they obtained generalized minimum aberration designs. They found that most of the generalized minimum aberration designs are projections of the 20-run Hadamard matrices and thus agrees with the designs reported in Deng and Tang (2002). However, they found the generalized minimum aberration designs for m = 6 and m = 7 are not projections of the Hadamard matrices. This agrees with the results from Li, Tang and Deng (2004) and Xu and Deng (2005). The complete catalogs of 12, 16 and 20 runs were later used by Li, Lin and Ye (2003) in the choice of optimal foldover plans, by Cheng, Li and Ye (2004) in the construction of blocked nonregular designs, by Loeppky, Bingham and Sitter (2006) for constructing nonregular robust parameter designs, and by Li (2006) for constructing screening designs for model selection.
Xu, Cheng and Wu (2004) considered the design issues related to the dual objectives of factor screening and interaction detection for quantitative factors. They proposed a set of optimality criteria to assess the performance of designs and a three-step approach to searching for optimal designs. They not only searched over all projection designs from the commonly used OA(18, 3 7 , 2) given by columns B to H in Table 2 and 68 saturated OA (27, 3 13 , 2) from Lam and Tonchev (1996), but also used an algorithm due to Xu (2002) to construct new designs directly. They presented many efficient and practically useful three-level nonregular designs with 18 and 27 runs for the dual objectives. Evangelaras et al. (2007) completely enumerated all nonisomorphic orthogonal arrays with 18 runs and 3 levels. Their results suggest that there are 4, 12, 10, 8, and 3 nonisomorphic OA(18, 3 m , 2) for m = 3, 4, 5, 6, and 7, respectively.
Loeppky, Sitter and Tang (2007) proposed to rank two-level orthogonal designs based on the number of estimable models containing a subset of main effects and their associated two-factor interactions. They argued that by ranking designs in this way, the experimenter can directly assess the usefulness of the experimental plan for the purpose in mind. They presented catalogs of useful designs with 16, 20, 24, and 28 runs. All these algorithmic constructions are limited to small run sizes (≤ 32) due to the existence of a large number of designs and the difficulty of determining whether two designs are isomorphic or equivalent. Katsaounisa and Dean (2008) gave a survey and evaluation of methods for determination of equivalence of factorial designs. Fang, Zhang and Li (2007) proposed an optimization algorithm for constructing generalized minimum aberration designs. It is not clear how effective their algorithm is for constructing large designs. Bulutoglu and Margot (2008) recently completely classified some orthogonal arrays of strength 3 up to 56 runs and of strength 4 up to 144 runs. However, these arrays have a small number of factors (≤ 11).

Optimality and Theoretical Results
A powerful tool in the study of regular designs is the complementary design technique. Every regular design can be determined by a unique complementary design. It is convenient to study the complementary design when it is small. Tang and Deng (1999) developed a complementary design theory for minimum G 2 -aberration nonregular designs and Xu and Wu (2001) further developed a theory for generalized minimum aberration designs. The theory was extended by Ai and Zhang (2004b) for blocked nonregular designs and by Ai and He (2006) for nonregular designs with multiple groups of factors, including robust parameter designs. However, unlike in the regular case, a nonregular design can have none, one or more than one complementary designs; therefore, the complementary design theory for nonregular designs is less useful than that for regular designs. Xu (2003) gave several sufficient conditions for a design to have minimum moment aberration and generalized minimum aberration among all possible designs. One sufficient condition is that for an orthogonal array of strength t its projection onto any t + 1 columns does not have repeated runs. For example, consider the OA(18, 3 6 , 2) given by columns C to H in Table 2. It is easy to verify that its projection onto any three columns does not have repeated runs. Thus, this design (and any of its projections) has minimum moment aberration and generalized minimum aberration among all possible designs. Another sufficient condition is that the numbers of coincidences between distinct rows are constant or differ by at most one. In other words, a design is optimal under the minimum moment aberration and generalized minimum aberration criteria if its design points are equally or nearly equally spaced over the design region. As an example, the OA(12, 2 11 , 2) given in Table 1 is optimal because the number of coincidences between any two distinct rows is 5. Generalizing this, Zhang et al. (2005) proposed a majorization framework and showed that orthogonality, aberration and uniformity criteria can be unified by properly choosing combinatorial and exponential kernels. Tang and Deng (2003) presented construction methods that yields maximum generalized resolution designs for 3, 4 and 5 factors and any run size N that is a multiple of 4. Butler (2003Butler ( , 2004 presented a number of construction results that allow minimum G 2 -aberration designs to be found for many of the cases with N =16, 24, 32, 48, 64 and 96 runs. Butler (2005) further developed theoretical results and presented methods that allow generalized minimum aberration designs to be constructed for more than two levels. A key tool used by Butler (2003Butler ( , 2004Butler ( , 2005 is some identities that link the generalized wordlength patterns with moments of the inner products or Hamming distances between the rows. These identities can be derived easily from the generalized Pless power moment identities developed by Xu (2003). Xu (2005a) constructed several nonregular designs with 32, 64, 128, and 256 runs and 7-16 factors from the Nordstrom and Robinson code, a well-known nonlinear code in coding theory. These designs are better than regular designs of the same size in terms of resolution, aberration and projectivity. By using linear programming he showed that 13 nonregular designs have minimum G 2 -aberration among all possible designs and seven orthogonal arrays have unique generalized wordlength patterns. Tang (2006) studied the existence and construction of orthogonal arrays that are robust to nonnegligible two-factor interactions. Butler (2007) showed that foldover designs are the only (regular or nonregular) two-level factorial designs of resolution IV or more for N runs and N/3 ≤ m ≤ N/2 factors. Yang and Butler (2007) studied two-level nonregular designs of resolution IV or more containing clear two-factor interactions and presented necessary and sufficient conditions for the existence of such designs. They gave many designs in concise grid representations for N = 48 up to 192 and N being a multiple of 16.
Stufken and Tang (2007) completely classified all two-level orthogonal arrays with t + 2 factors, strength t and any run size. The key tool they used is the theory of J-characteristics developed by Tang (2001). Cheng, Mee and Yee (2008) studied the construction of second-order saturated orthogonal arrays of strength three OA(N, 2 m , 3), which allows N − m − 1 two-factor interactions to be estimated besides m main effects.

Nonregular designs constructed via quaternary codes
The construction of large regular designs is known to be very difficult (Xu 2007). The problem is even harder for nonregular designs. The construction via quaternary codes is relatively straightforward and can generate good large nonregular designs.
A quaternary code is a linear subspace over Z 4 = {0, 1, 2, 3} (mod 4), the ring of integers modulus 4. A surprising breakthrough in coding theory is that many famous nonlinear codes such as the Nordstrom and Robinson code can be constructed via quaternary codes (Hammons et al. 1994). A key device is the so-called Gray map: φ : 0 → (0, 0), 1 → (0, 1), 2 → (1, 1), 3 → (1, 0), which maps each symbol in Z 4 to a pair of symbols in Z 2 . Let G be a k × n matrix and let C consist of all possible linear combinations of the row vectors of G over Z 4 . Applying the Gray map to C, one obtains a 4 k × 2n matrix or a two-level design, denoted by D. Although C is linear over Z 4 , D may or may not be linear over Z 2 .
Xu and Wong (2007) described a systematic procedure for constructing nonregular designs from quaternary codes. They first generated a k × (4 k − 2 k )/2 generator matrix G which has the following properties: (i) it does not have any column containing entries 0 and 2 only and (ii) none of its column is a multiple of another column over Z 4 . Xu and Wong (2007) showed that the binary image D generated by G is a 4 k × (4 k − 2 k ) design with resolution 3.5 whereas regular designs of the same size have resolution 3. To obtain designs with less than 4 k − 2 k columns, they developed a sequential algorithm, similar to those by Chen, Sun and Wu (1993) and Xu (2005b). They also presented a collection of nonregular designs with 32, 64, 128 and 256 runs and up to 64 factors, many of which are better than regular designs of the same size in terms of resolution, aberration and projectivity. Phoa and Xu (2008) further investigated the properties of quarter-fraction designs which can be defined by a generator matrix that consists of an identity matrix plus an extra column. They showed that the resolution, wordlength and projectivity can be calculated in terms of the frequencies of the numbers 1, 2 and 3 that appear in the extra column. These results enabled them to construct optimal quarter-fraction designs via quaternary codes under the maximum resolution, minimum aberration and maximum projectivity criteria. These designs are often better than regular designs of the same size in terms of the design criterion. The generalized minimum aberration designs constructed via quaternary codes have the same aberration as the minimum aberration regular designs, and frequently with larger resolution and projectivity. A maximum projectivity design is often different from a minimum aberration or maximum resolution design but can have much larger projectivity than a minimum aberration regular design. They further showed that some of these designs have generalized minimum aberration and maximum projectivity among all possible designs.
There are two obvious advantages of using quaternary codes to construct nonregular designs: (1) relatively straightforward construction and (2) simple design representation. Since the designs are constructed via linear codes over Z 4 , one can use column indexes to describe these designs. The linear structure of a quaternary code also facilitates the derivation and analytical study of properties of nonregular designs.

Concluding Remarks and Future Directions
We have discussed recent developments in nonregular fractional factorial designs in the preceding sections. In a nutshell, when we compare regular designs with nonregular designs, nonregular designs have the following advantages: Some of the disadvantages of nonregular designs are that they are more complicated to analyze and some estimates of factorial effects may have larger variance than others.
This review does not include the developments in supersaturated designs, which are factorial designs whose run sizes are not enough for estimating all the main effects. The research on supersaturated designs has been very active since the influential work of Lin (1993) and Wu (1993). Broadly speaking, supersaturated designs are nonregular designs and optimality criteria such as generalized resolution and generalized minimum aberration can also be applied directly. As mentioned earlier, Xu (2003) developed a unified theory for nonregular and supersaturated designs using the concept of minimum moment aberration. Xu and Wu (2005) obtained more theoretical results under the generalized minimum aberration criterion for multi-level and mixed-level supersaturated designs. Gilmour (2006) reviewed the recent development of two-level supersaturated designs for factor screening.
Finally we highlight some future directions of research for nonregular designs and comment briefly why we feel they are useful: 1. applications of nonregular designs 2. analysis of nonregular designs 3. construction of good nonregular designs with large run sizes 4. optimality results with respect to the generalized resolution.
Despite significant developments in recent years and the advantages of using nonregular designs, they are still widely used for screening main effects only in practice and applications are largely limited to industry. We hope that by documenting recent advances in nonregular designs, our work may stimulate greater research interest in nonregular designs. We feel that there are opportunities that nonregular designs can be effectively applied to other fields to reduce experimental cost and gain improvement in statistical efficiency.
The analysis of nonregular designs requires more attention. Although one can use any general purpose variable selection procedures, it is desirable to have user-friendly packages that incorporate the special features of nonregular designs in the analysis. More analysis strategies and comparisons are needed to further understand and utilize the complex aliasing structure of nonregular designs.
There are plenty of catalogs of optimal nonregular designs with small run sizes (≤ 32). With the popularity of computer experiments, more and more large nonregular factorial designs will be used in practice. Mee (2004) illustrated how nonregular designs can be used to reduce run sizes significantly in applications requiring estimation of the main effects and two-factor interactions for a large number of factors. He suggested the use of a 2,096-run nonregular design with 47 factors as an alternative to a 4,096-run regular designs in a ballistic missile simulation application. The quaternary code construction method is very promising in this regard and is able to produce large nonregular designs with good properties. Indeed, this 2,096-run nonregular design can be conveniently constructed via quaternary codes.
Several optimality results and theories have been obtained for the minimum G 2 -aberration and the generalized minimum aberration. However, at the present time, there is very limited results on the generalized solution for nonoregular designs. This is useful because it is always helpful to know whether a design is close to the optimal design or not, without knowing the optimal design.