EMPIRICAL COMPARISON OF THE MULTIDIMENSIONAL MODELS OF ITEM RESPONSE THEORY IN E-COMMERCE

Tezza, Rafael; Bornia, Antonio Cezar; Spenassato, Débora; Trierweiller, Andréa Cristina

doi:10.1590/0101-7438.2016.036.03.0503

Rafael Tezza ^**Corresponding author.

Centro de Ciências da Administração e Socioeconômicas, Universidade do Estado de Santa Catarina, Av. Madre Benvenuta, 2037, 88035-001 Itacorubí, Florianópolis, SC, Brasil. E-mail: rafael.tezza@udesc.br

Antonio Cezar Bornia

Departamento de Engenharia de Produção e Sistemas, Universidade Federal de Santa Catarina, Caixa Postal 476, Campus Universitário, 88010-970 Florianópolis, SC, Brasil. E-mail: cezar.bornia@ufsc.br

Débora Spenassato

Instituto de Matemática, Estatística e Física, Universidade Federal do Rio Grande, Campus Carreiros, Av. Itália Km 8, 96201-900 Rio Grande, RS, Brasil. E-mail: debospenassato@gmail.com

Andréa Cristina Trierweiller

Departamento de Tecnologias da Informação e Comunicação, Universidade Federal de Santa Catarina, Rua Prefeito Walter Belinzoni, 219, 88900-017 Araranguá, SC, Brasil. E-mail: andreatri@gmail.com

*Corresponding author.

Analysis	Target
Tetrachoric correlation matrix	Evaluation of the number of dimensions
Parallel analysis	Evaluation of the number of dimensions
Factor analysis method of restricted information	Evaluation of the number of dimensions and Analysis dimensional structure
Factor analysis method of full information	Evaluation of the number of dimensions and Analysis dimensional structure
Unidimensional model of two parameters IRT	Assessment of the adequacy of the unidimensional structure and multiple unidimensional structure
Multidimensional model of two parameters compensatory MIRT	Assessment of the adequacy of the multidimensional structure of IRT
Bifactorial model IRT	Assessment of the adequacy of the Bifactorial model IRT

Dimension	1	2	3	4	5	6	7	8	9	10
Eigenvalue	11.0	5.7	4.4	3.9	3.4	3.1	2.7	2.5	2.3	2.2
Accumulated proportion of explained variation	14.6	22.2	28.0	33.3	37.8	41.9	45.5	48.8	51.9	54.8

	χ²	Degrees of liberty	p-value	AIC	BIC
Mod1				23049.49	23648.21
Mod2	542.489	74	<0.001	22655.00	23549.09
Mod3	335.394	73	<0.001	22465.61	23651.07
Mod4	219.207	72	<0.001	22390.40	23863.25
Mod5	138.058	71	<0.001	22394.34	24150.59
Mod6	-1848.45	70	<0.005	24382.79	26418.44
Mod7	47.51	69	0.9775	24473.28	26784.34
Mod8	25.239	68	0.9999	24584.04	27166.52

Item	Parameters
	Discrimination	Standard Error	Difficulty	Standard Error
57	1.77	0.26	1.16	0.13
58	6.68	3.79	0.80	0.20
60	1.93	1.01	-2.64	0.67
69	0.64	0.18	-1.52	0.39
71	1.00	0.29	-2.10	0.46
74	8.91	58.30	0.58	0.83
75	0.84	0.18	-0.59	0.16

Item	a1	Se	a2	se	a3	se	a4	se	MDISC	d	se	MDIFF
03	0.30	1.03	1.52	3.50	0.07	39.49	-0.040	54.53	1.55	4.07	1.24	-2.62
05	-0.32	22.88	0.28	31.36	-0.40	34.43	-0.880	20.25	1.06	5.23	9.19	-4.95
06	1.09	1.70	1.91	15.07	0.30	54.56	-0.250	64.98	2.23	4.62	1.76	-2.07
08	-0.14	0.59	0.15	39.66	-0.41	35.75	-1.210	16.01	1.29	0.81	0.32	-0.63
10	-0.12	0.67	0.65	41.81	0.06	19.93	1.140	23.05	1.32	2.08	0.53	-1.58
12	-0.13	0.78	1.03	42.84	0.87	42.55	-0.680	23.18	1.52	2.49	0.58	-1.64
19	0.67	0.81	0.90	28.51	0.51	14.99	0.920	23.47	1.54	1.46	0.48	-0.95
21	1.43	1.46	0.89	38.57	1.89	15.48	0.970	30.04	2.71	5.12	2.41	-1.89
22	0.56	0.98	0.82	32.96	0.57	34.70	-0.590	19.83	1.29	3.75	1.02	-2.91
23	0.55	0.90	1.08	18.07	0.59	17.63	0.610	28.72	1.48	3.55	0.66	-2.40
24	11.44	177.69	5.62	257.53	16.62	283.21	7.090	291.11	22.11	46.63	719.38	-2.11
25	0.39	1.62	-0.11	99.18	2.82	24.08	-0.980	79.36	3.01	5.57	2.82	-1.85
27	0.06	0.56	0.62	40.23	-0.16	17.11	1.010	26.64	1.20	-0.60	0.34	0.50
28	0.15	0.72	0.04	52.30	0.42	41.15	1.560	10.25	1.62	-1.35	0.44	0.83
29	0.47	0.68	0.37	49.02	-0.36	23.97	1.150	22.52	1.34	-2.03	0.46	1.51
30	0.76	0.88	1.01	39.26	0.78	20.63	1.290	23.39	1.97	1.35	0.58	-0.69
32	0.62	0.65	0.65	18.39	0.65	10.41	0.610	14.00	1.27	1.58	0.42	-1.25
33	0.92	1.29	2.29	32.29	1.24	58.78	-0.010	61.27	2.76	5.55	2.87	-2.01
35	0.10	0.45	0.64	22.48	0.10	10.49	0.640	21.77	0.92	-0.35	0.30	0.38
37	0.62	0.77	1.45	11.39	-0.10	31.66	0.260	56.15	1.60	2.36	0.56	-1.47
38	0.95	0.70	0.47	33.23	0.70	21.06	1.100	10.75	1.68	0.67	0.48	-0.40
40	0.15	0.50	0.90	17.42	0.05	14.72	0.490	32.63	1.04	-0.15	0.28	0.14
43	0.01	0.86	1.06	9.27	-0.18	23.78	0.160	43.42	1.09	-4.20	1.18	3.86
45	0.28	0.66	1.13	25.77	0.05	17.52	0.710	40.90	1.36	1.49	0.41	-1.09
46	0.53	0.77	0.49	32.04	1.32	19.01	1.02	22.69	1.82	1.24	0.50	-0.68
47	0.60	3.15	1.07	48.33	1.16	41.31	-0.58	22.02	1.78	5.19	3.09	-2.91
48	0.90	1.04	1.62	20.43	0.79	28.94	0.67	44.95	2.12	3.06	0.80	-1.44
52	0.61	0.64	0.47	19.32	0.95	7.59	0.46	14.58	1.31	1.73	0.44	-1.32
55	0.45	0.92	0.23	41.62	1.16	16.83	-0.43	24.72	1.34	4.40	0.87	-3.29
56	1.91	4.34	1.81	18.87	0.93	41.42	0.33	49.98	2.81	6.88	16.99	-2.45
57	-2.08	0.72	-0.14	30.90	1.17	2.99	-0.03	35.93	2.39	-2.41	0.68	1.01
58	-50.47	398.55	8.56	309.51	6.92	181.90	10.11	217.53	52.64	-41.73	332.82	0.79
59	0.38	0.99	-1.19	27.54	1.34	46.79	0.69	76.93	1.96	3.80	1.45	-1.94
60	-2.74	4.63	-0.46	70.45	0.05	61.38	1.92	21.13	3.38	7.25	13.39	-2.15
61	-0.75	0.53	-0.40	21.09	0.95	25.62	0.62	38.81	1.42	0.38	0.28	-0.27
64	0.71	1.75	-1.92	38.71	1.75	56.92	0.33	115.42	2.71	6.39	3.54	-2.36
65	0.26	3.79	-2.53	56.83	2.89	84.56	0.98	160.92	3.97	8.67	20.84	-2.18
66	1.01	1.19	-0.14	27.72	0.74	29.10	0.94	24.34	1.57	4.87	2.89	-3.10
69	-0.86	0.56	0.32	12.34	-0.47	18.09	-0.40	23.44	1.11	1.11	0.37	-1.004
70	-0.35	0.44	-0.10	17.00	0.82	13.87	0.43	25.20	1.00	-0.57	0.27	0.573
71	-1.02	0.56	0.46	9.67	0.46	7.56	0.25	9.91	1.24	2.23	0.39	-1.81
73	0.82	11.08	-3.37	70.10	2.36	86.72	-0.22	185.21	4.20	10.71	37.53	-2.55
74	-4.25	1.69	0.60	25.59	1.23	10.37	0.62	19.79	4.51	-3.05	1.27	0.68
75	-0.79	0.37	0.29	15.64	0.33	13.21	-0.24	6.18	0.94	0.51	0.23	-0.54

Item	General	a1	a2	a3	a4	d	Item	General	a1	a2	a3	a4	d
3	0.531	0	0.374	0	0	2.116	43	0.277	0	2.097	0	0	-4.691
5	-0.483	0	0	0	-0.178	-0.178	45	0.627	0	0.274	0	0	0.793
6	0.755	0	0.676	0	0	2.342	46	0.999	0	0	0.584	0	0.69
8	-0.447	0	0	0	-0.839	0.538	47	0.672	0	0	0.402	0	2.591
10	0.522	0	0	0	0.747	1.293	48	1.089	0	0.722	0	0	1.79
12	0.106	0	0.831	0	0	1.418	52	0.709	0	0	0.253	0	0.956
19	0.948	0	0	0	0.021	0.832	55	0.388	0	0	0.741	0	2.519
21	1.804	0	0	1.104	0	3.364	56	1.31	0	0.321	0	0	3.426
22	0.233	0	0.73	0	0	2.2	57	-0.203	-1.064	0	0	0	-1.209
23	0.79	0	0.333	0	0	2.03	58	-0.513	-3.581	0	0	0	-2.966
24	6.115	0	0	4.418	0	12.924	59	0.243	0	0	0.9	0	1.925
25	0.385	0	0	0.997	0	2.252	60	-0.045	-1.175	0	0	0	2.986
27	0.482	0	0	0	0.151	-0.332	61	0.102	0	0	0.495	0	0.161
28	0.632	0	0	0	0.567	-0.768	64	0.12	0	0	1.796	0	3.903
29	0.505	0	0	0	0.218	-1.074	65	0.102	0	0	2.676	0	5.447
30	1.286	0	0	0	0.428	0.822	66	0.69	0	0	0	-0.155	2.572
32	0.805	0	0	0.068	0	0.916	69	-0.324	-0.387	0	0	0	0.616
33	1.146	0	1.127	0	0	3.182	70	0.228	0	0	0.411	0	-0.337
35	0.456	0	0.412	0	0	-0.233	71	0.051	-0.635	0	0	0	1.253
37	0.597	0	0.648	0	0	1.317	73	-0.103	0	-0.694	0	0	3.168
38	0.953	0	0	0	0.151	0.353	74	-0.253	-2.751	0	0	0	-1.829
40	0.519	0	0.076	0	0	-0.103	75	-0.091	-0.509	0	0	0	0.296

Model	Log of maximum verisimilitude	AIC	BIC	RMSEA	M2
Unidimensional Model	13826	14002.7	14362.6	0.07	2844.39
MIRT Model	12616	13016.1	13833.9	0.04	1405.22
Bifactorial model	13200	13464.1	14003.8	0.05	1937.95

Website	User orientation during navigation	Accessibility and reliability of the system	User control or user interaction with the system	Presentation of information
001	0,105	0,130	-0,864	0,677
002	0,367	0,524	-0,753	0,040
003	-0,887	-0,328	-0,557	0,525
004	0,782	0,324	-0,879	0,314

Item	Description of the item	Source
01	When opening the homepage do pop-up windows open?	^{Storey et al. (2002}93 STOREY M-A, PHILLIPS B, MACZEWSKI M & WANG M. 2002. Evaluating the usability ofWeb-based learning tools. Educational Technology & Society, 5: 91-100.); ^{Petre et al. (2006}76 PETRE M, MINOCHA S & ROBERTS D. 2006. Usability Beyond the Website: an Empirically Grounded E-commerce Evaluation Instrument for the Total Customer Experience. Behaviour & Information Technology, 25: 189-203.); ^{Tezza et al. (2011}97 TEZZA R, BORNIA AC & ANDRADE DF. 2011. Measuring web usability using item response theory: Principles, features and opportunities. Interacting with Computers, 23: 167-175.)
02	Does the homepage present the site’s main content areas (navigation)?	^{Nielsen (2000}66 NIELSEN J. 2000. Projetando websites. Gulf Professional Publishing.)
03	Does the homepagemake clearwhat the site does (demonstrate the main products and or a brief description of its objective and or benefits that it offers), without needing the roll-bar?	^{Nielsen (2000}66 NIELSEN J. 2000. Projetando websites. Gulf Professional Publishing.)
04	Does the homepage present a summary of the most important sales?	^{Nielsen (2000}66 NIELSEN J. 2000. Projetando websites. Gulf Professional Publishing.)
05	Does the site have moving images that can distract the user?	^{Colesca (2007}25 COLESCA SE. 2007. An Assessment of the Quality of the Romanian Urban Web Sites. Informatica Economică, 42: 26-33.)
06	Do the links for sales go directly to the sale announced?	^{Nielsen (2000}66 NIELSEN J. 2000. Projetando websites. Gulf Professional Publishing.)
07	Are the menus in alphabetical order?	^{Olsina et al. (2001}72 OLSINA L, GODOY D, LAFUENTE GJ & ROSSI G. 1999. Specifying Quality Characteristics and Attributes for Websites, in Proceedings of the First ICSE Workshop on Web Engineering, 16-17 May Los Angeles, USA.)
08	Does the site have a cascademenu?	^{Nielsen & Loranger (2006}68 NIELSEN J & LORANGER H. 2006. Prioritizing web usability. Pearson Education.)
09	For navigation, is there a track on the left or links on top of the page?	^{Nielsen & Tahir (2002}69 NIELSEN J & TAHIR M. 2002. Homepage usability: 50 websites deconstructed. Indianapolis, In: New Riders.)
10	Are sub-categories grouped?	^{Nielsen & Tahir (2002}69 NIELSEN J & TAHIR M. 2002. Homepage usability: 50 websites deconstructed. Indianapolis, In: New Riders.)
11	Does the title of the window (browser) list the name of the site in the first place?	^{Nielsen & Tahir (2002}69 NIELSEN J & TAHIR M. 2002. Homepage usability: 50 websites deconstructed. Indianapolis, In: New Riders.)
12	Are there information for telephone contact or an address?	^{Nielsen (2000}66 NIELSEN J. 2000. Projetando websites. Gulf Professional Publishing.); ^{Nielsen & Tahir (2002}69 NIELSEN J & TAHIR M. 2002. Homepage usability: 50 websites deconstructed. Indianapolis, In: New Riders.);
13	Do the page have a consistent visual appearance, that is do they always have the same visual appearance?	^{Aladwani & Palvia (2002}8 ALADWANI AM & PALVIA PC. 2002. Developing and validating an instrument for measuring user-perceived web quality. Information & Management, 39: 467-476.); ^{Ferreira & Nunes (2008}33 FERREIRA SBL & NUNES RR. 2008. E-Usabilidade, Rio de Janeiro: LTC.)
14	Does the site have the option to access it in other languages?	^{Colesca (2007}25 COLESCA SE. 2007. An Assessment of the Quality of the Romanian Urban Web Sites. Informatica Economică, 42: 26-33.)
15	Do the clickable words (in color or underlined) have a distinct form when selected?	^{Nielsen & Loranger (2006}68 NIELSEN J & LORANGER H. 2006. Prioritizing web usability. Pearson Education.)
16	Do the field labels begin with a capital letter, and are the remaining letters in lower case?	^{Aldwani & Palvia (2002}8 ALADWANI AM & PALVIA PC. 2002. Developing and validating an instrument for measuring user-perceived web quality. Information & Management, 39: 467-476.)
17	Are the titles aligned to the left?	^{Inkpen et al. (2006}47 INKPEN K, DEARMAN D & ARGUE R. 2006. Left-Handed Scrolling for Pen-Based Devices. International Journal of Human-Computer Interaction, 21: 91-108.)
18	Are text paragraphs separated?	^{Zerfass & Hartmann (2005}108 ZERFASS A , HARTMANN B. 2005. The usability factor: Improving the quality of E-content. In E-Content (pp. 165-182). Springer Berlin Heidelberg.)
19	Are apparently clickable words in fact clickable?	^{Nielsen (2000}66 NIELSEN J. 2000. Projetando websites. Gulf Professional Publishing.)
20	Are the title for screens, windows and dialog boxes on top, centered or left justified?	^{Inkpen et al. (2006}47 INKPEN K, DEARMAN D & ARGUE R. 2006. Left-Handed Scrolling for Pen-Based Devices. International Journal of Human-Computer Interaction, 21: 91-108.)
21	Do all the pages have a search field?	^{Nielsen (2000}66 NIELSEN J. 2000. Projetando websites. Gulf Professional Publishing.)
22	When there is rolling, are there design elements (in the initial screen) thaat appear with end of page markers?	^{Nielsen & Loranger (2006}68 NIELSEN J & LORANGER H. 2006. Prioritizing web usability. Pearson Education.)
23	Does the company logo in the upper left corner on all the site pages?	^{Roth et al. (2010}87 ROTH S, SCHMUTZ P, PAUWELS S, BARGAS-AVILA J & OPWIS K. 2010. Mental models for web objects: where do users expect to find the most frequent objects in online shops, news portals, and company web pages? Interacting with Computers, 22: 140-15.)
24	Is there a link with a single click that leads to the homepage?	^{van der Merwe & Bekker (2003}99 VAN DER MERWE R & BEKKER J. 2003. A framework and methodology for evaluating e-commerce Web sites. Internet Research: Electronic Networking Applications and Policy, 13: 330-341.)
25	Does the site allow navigating its pages in only one window, that is, there is not opening of new windows in amid the navigation?	^{Nielsen & Loranger (2006}68 NIELSEN J & LORANGER H. 2006. Prioritizing web usability. Pearson Education.); ^{Tezza et al. (2011}97 TEZZA R, BORNIA AC & ANDRADE DF. 2011. Measuring web usability using item response theory: Principles, features and opportunities. Interacting with Computers, 23: 167-175.)
26	Are there different colors for previously visited links?	^{Nielsen & Tahir (2002}69 NIELSEN J & TAHIR M. 2002. Homepage usability: 50 websites deconstructed. Indianapolis, In: New Riders.)
27	Is there a list of frequently asked questions - FAQs?	^{Colesca (2007}25 COLESCA SE. 2007. An Assessment of the Quality of the Romanian Urban Web Sites. Informatica Economică, 42: 26-33.); ^{Hernández et al. (2007}43 HERNÁNDEZ B, JIMÉNEZ J & MARTÍN MJ. 2009. Key website factors in e-business strategy. International Journal of Information Management, 29: 362-371.); ^{Tezza et al. (2011}97 TEZZA R, BORNIA AC & ANDRADE DF. 2011. Measuring web usability using item response theory: Principles, features and opportunities. Interacting with Computers, 23: 167-175.)
28	When entering search terms in the search field does the search engine offer suggestions?	^{Long & McMellon (2004}62 LONG M & MCMELLON C. 2004. Exploring the determinants of retail service quality on the internet. Journal of Services Marketing, 18: 78-90.)
29	Is the search system flexible in relation to the terms used by the user, that is, if the user types a term incorrectly, does the search system suggest a correction?	^{Agner (2012}5 AGNER L. 2012. Ergondesign e arquiterura de informação: trabalhando com o usuário. Rio de Janeiro: Quartet, 3a. edição.)
30	Do the search results allow classification by other criteria in addition to cost?	^{Nielsen (2000}66 NIELSEN J. 2000. Projetando websites. Gulf Professional Publishing.); ^{Tezza et al. (2011}97 TEZZA R, BORNIA AC & ANDRADE DF. 2011. Measuring web usability using item response theory: Principles, features and opportunities. Interacting with Computers, 23: 167-175.)
31	Do long lists present indicators of communication, number of items and pages?	^{Nielsen & Loranger (2006}68 NIELSEN J & LORANGER H. 2006. Prioritizing web usability. Pearson Education.); ^{Tezza et al. (2011}97 TEZZA R, BORNIA AC & ANDRADE DF. 2011. Measuring web usability using item response theory: Principles, features and opportunities. Interacting with Computers, 23: 167-175.)
32	Are page continuation items visible?	^{Agner (2012}5 AGNER L. 2012. Ergondesign e arquiterura de informação: trabalhando com o usuário. Rio de Janeiro: Quartet, 3a. edição.)
33	Is the price of a product next to the image or link for the product?	^{Hung & McQueen (2004}45 HUNG W-H & MCQUEEN RJ. 2004. Developing an Evaluation Instrument for eCommerce Web Sites from the First-Time Buyer’s Viewpoint. Electronic Journal of Information Systems Evaluation, 7: 31-42.)
34	Is it possible to expand the photos of products to visualize details?	Novikova (2009); ^{Nielsen & Loranger (2006}68 NIELSEN J & LORANGER H. 2006. Prioritizing web usability. Pearson Education.); ^{Tezza et al. (2011}97 TEZZA R, BORNIA AC & ANDRADE DF. 2011. Measuring web usability using item response theory: Principles, features and opportunities. Interacting with Computers, 23: 167-175.)
35	In products in which there is more than one perspective, is it possible to visualize all the perspectives?	^{Nielsen & Tahir (2002}69 NIELSEN J & TAHIR M. 2002. Homepage usability: 50 websites deconstructed. Indianapolis, In: New Riders.); ^{Tezza et al. (2011}97 TEZZA R, BORNIA AC & ANDRADE DF. 2011. Measuring web usability using item response theory: Principles, features and opportunities. Interacting with Computers, 23: 167-175.)
36	Are the groups of command buttons available in the column to the right, or on a line below the objects to which they are associated?	^{Inkpen et al. (2006}47 INKPEN K, DEARMAN D & ARGUE R. 2006. Left-Handed Scrolling for Pen-Based Devices. International Journal of Human-Computer Interaction, 21: 91-108.)
37	Is there sufficient information about the products (size, basic characteristics)?	^{Signore (2005}91 SIGNORE O. 2005. ‘A comprehensive model for Web sites quality’. Proc. of the Seventh IEEE International Symposium on Web Site Evolution (WSE’05), pp. 30-36.)
38	Is there a way for consumers to provide to insert feedback about the products?	^{Agner (2012}5 AGNER L. 2012. Ergondesign e arquiterura de informação: trabalhando com o usuário. Rio de Janeiro: Quartet, 3a. edição.)
39	Are there background images for the text?	^{Nielsen & Tahir (2002}69 NIELSEN J & TAHIR M. 2002. Homepage usability: 50 websites deconstructed. Indianapolis, In: New Riders.)
40	Does the site present related products at the end of the page?	^{Nielsen (2000}66 NIELSEN J. 2000. Projetando websites. Gulf Professional Publishing.)

Brasil

Brasil

EMPIRICAL COMPARISON OF THE MULTIDIMENSIONAL MODELS OF ITEM RESPONSE THEORY IN E-COMMERCE

ABSTRACT