Which methodology to use bearing in mind subject matter

Open discussion
#1

Hi everyone,

I’m doing an online MSc in Computer Science with Cyber Security. As it’s online there’s no face-to-face with any of the tutors. I’m currently in the module before the research project where I’m preparing the research project proposal (RPP). I’m having real trouble trying to get straight answers from my tutors, which is infuriating!

I barely scraped by the research methods module but I’m still here, (barely) near the end of the course.
The area that I’d like to do my project in is around Information Security Management Systems, so this is not at the programming end of things and is more the intersection of IT and management.

When I’ve been doing review of work in this area, I can see that in some cases the authors are producing a framework / excel sheet / some other artefact and there is no evaluation of this artefact. I can understand that for space reasons sometimes the evaluations aren’t included in some articles but I’m still left scratching my head as to what type of methodology has been used.

This is a really basic question, for the project, can I produce any kind of artefact/hypothesis and then evaluate it?

The second part of this is related to the evaluation. I’m imagining that in a management area like this, then the evaluations will be more qualitative than quantitative?

#2

Hi @SpanishTony!! Welcome to the forum!!

I definitely can relate with your frustration of wanting answers from your advisors and getting any. However, keep in mind that this is YOUR project and not theirs. So I kind of understand their take on not giving you any directive answer. On this topic, I recommend you investigate what you want to do, build a rational decision for why you want to pick the given approach, develop what you can, and show them to pick their opinion. Do first, ask questions later. This way they will opine on something more concrete rather than possibilities.

I recently dropped out of a Management program and now I am applying for Organizational Development programs. So I believe our fields have some similarities such that I believe I understand your questions about the papers feeling kind of incomplete. From my experience with the Management literature, I have always seen studies using or even assessing the quality of measures without presenting them in the actual paper. And I have never seen anyone mentioning anything about page limits for it.

On your comments about the evaluation of the framework, I assume that the publication of the framework is some kind of its assessment because the paper is (probably, so I recommend you to check this) peer-reviewed. About the methodology being used, this is something I have been scratching my head for a very long time in my field, so if the process is the same (which I believe it probably is) the methodology for building it is simple reasoning coming from the literature review used and probably presented in the paper. In my field, I always felt it was some kind of black box that no one talked about but I recently discovered that it is way simpler than I thought. This may be your case too.

Another option for you is simply to send the authors of the papers you read an email asking them that question. I have been pulling this “crazy” move and everybody has been more than happy to share their work and thoughts about their work!

This is a really basic question, for the project, can I produce any kind of artefact/hypothesis and then evaluate it?
Now that is the million-dollar question!! Yes and no. Can you do it? Yes, you can. It is your study, so as long as you keep the rigor from the research field, you can do anything you want. But let’s separate your question into two parts here.

When it comes to creating any possible hypothesis, then yes. You sure can. As long as you are able to (1) review the existing theory about the topics you want to talk about, and (2) build a rationale that would support your hypothesis, then that’s it. Go for it.

Now when it comes to creating an “artifact” the idea that comes to my mind is creating a measure, a set of items/questions destined to quantitatively measure the construct you created. Can you do it? Yes, you can. Should you do it? I would not recommend it because doing it in the proper way takes time and effort and I do not believe you should focus on it in your Master’s dissertation.

However, considering it is a master thesis rather than an actual paper to be published, your advisors may give you some leeway about the rigor of your measure.

P.s.: Sorry for giving you the f%cked up answer “it depends” but in the end it really does… :sweat_smile:

To understand more about designing measures I highly recommend you the book Salkind, N. J. (2017). Tests & Measurement for People Who (Think They) Hate Tests & Measurement (3rd edition). SAGE Publications, Inc. You can find a copy on the internet. Chapters 3 to 6 will give you a general idea of what you need to validate a measure you create. It is a very easy reading so the return over your time invested will definitely be paid.

Before saying anything about the second part, what do you mean by “framework / excel sheet / some other artifact”? Because when you say excel sheet and artifact what comes to mind is some kind of measure (questions to measure a construct as mentioned before) but when you say framework it comes to mind a model showing a relationship between construct A and construct B.

If it is about the measure then you are right. It is qualitative and it is called content validity. If it is about a model showing the relationship between 2 constructs then the optimal scenario would be to have an empirical study showing that relationship. Sometimes that is not possible for a myriad of reasons then that would be a conceptual paper. If that is the case, the evaluation is qualitative through logical reasoning laid out in the paper itself. The goal, in this case, is (should be) to put the idea out there, generate a conversation, and in the future create a measure to empirically identify this relationship.

I hope I have helped you man!! I know how it is to try to do something in this messy world of social science without any support. For instance, most of the things I shared with you here I just learned on my own in the past few months.

Let me know if I can help you somehow!!

Best,
Diego Tavares

#3

Wow thanks for taking the time to reply! There was really a lot of information to take on board. Addressing some of the points your raised:

Yes, that makes a lot of sense, it also seems churlish of me to complain about a tutor when things can be so easily misunderstood.

Again, a lot of academia I’ve found to be complicated descriptions of some simple ideas (I guess because we understand things through context which isn’t scientific).

Yes! that’s so obvious but I hadn’t thought of it before. I will try it tonight.

My current thinking is to produce some artefact and then get it reviewed through a survey by a bunch of InfoSec specialists. Do I have a group of InfoSec specialists to hand? No, but I suppose I’ll have to find a group because generally speaking I don’t think that for InfoSec data, it’s not so easy to find data that can be used in a research project (because of it’s nature plus commercial reasons etc.)

I’ll see if I can find the book you mentioned in our Library.

There is a lot here! I’m still thinking about my subject, if I present it one way I could maybe show a relationship between two elements or I could think about the ‘thing’ a different way and it could be used as a measure? This is probably why I’m getting confused.

The course have a book in their book list which I’ve been reading and it’s been very useful C.W Dawson. *Projects in computing and information systems: a student’s guide. As it suggests approaches based on whether you are programming or on the other side, like myself, on information security.

Thanks, you’ve been a great help, there’s a lot for me to think about and hopefully I can come back with maybe some better questions.

Best,
Anthony

#4

@SpanishTony Don’t mention it!! I can definitely relate to what you are going through because I had similar doubts but during the first semester of a Ph.D. and I had no one to answer because, at least in my opinion, the program assumed everybody knew how to do research. In addition, for my master’s thesis, I did a qualitative study from a consulting perspective and the PhD program focused only on quantitative research. So I spent the whole semester with similar questions but no one to answer them. :sweat_smile: :ok_hand:

If I’m going to be honest, I do not think it was churlish on your part. Unfortunately, it is more often than not finding advisors that may be experts in their field but know nothing about relating with others in the work environment. In my opinion, if I can find that in an Organizational Behavior Ph.D. program that is focused on Leadership, everybody can find bad advisors too. I believe this kind of guidance should have been told to you upfront in the clearest way possible so you do not raise any kind of misaligned expectations with reality just like you did. They are the experts in this process. Not you.

Unfortunately, that is also the norm in the Management literature but not, as far as I have read, in the Organizational Development literature. My opinion is that this is more common in fields with a scant academy-practice relationship. My suggestion to you unfortunately is to plow through it. At some moment you will get the hang of it. You can also try to find review papers about the topic you are interested in because they show a bit more concern about explaining the concepts. For instance, when studying Social Networks I had to read 3 review papers in parallel because some concepts were more detailed in one than the others and I found that more productive rather than reading one after the other.

From your statement, your concept of artifact is some form of measurement. Personally, I find it interesting to have this kind of learning out of a master’s but you have to keep in mind how rigorous your advisor(s) will be about this. I have this study as a reference for measure design ( Stilwell, R. A., Pasmore, W. A., & Shon, D. (2016). Change Leader Behavior Inventory: Development and Validation of an Assessment Instrument. The Journal of Applied Behavioral Science, 52(4), 373–395. https://doi.org/10.1177/0021886316663406) and they use 15 experts to assess how accurate the items were. Maybe you can reach out to the authors of the papers you like and ask them if they can review your items once you design them.

If you want to find a relationship between two elements, keep in mind that you will probably have to learn Statistics and some programming languages such as R or Python. In contrast, when you design a measure, as you explained before, as far as I know, its weight is more on the step-by-step process and less on the statistics side. Personally, I believe this is more interesting because it is more out of the ordinary because any researcher can find a relationship between two elements but not everyone knows how to effectively design a measure. But that is my bias!! I do not want to negatively influence you on doing something on your project you do not want to!!

In addition, if you wish to pursue the relationship aspect between two elements, I highly recommend you the following books:

  • Salkind, N. J., & Frey, B. B. (2020). Statistics for people who (think they) hate statistics (Seventh edition). SAGE.
  • Hui, E. G. M. (2019). Learn R for Applied Statistics: With Data Visualizations, Regressions, and Statistics. Apress. Learn R for Applied Statistics | SpringerLink
  • Salkind, N. J., & Shaw, L. A. (2019). Statistics for People Who (Think They) Hate Statistics Using R. SAGE Publications.

(Salkind & Frey, 2020) is THE book for statistics for basic research. Simple, easy comprehension, straightforward, easy vocabulary, the whole package. What I hated during the first semester of a PhD program, I learned to love through this book.
(Hui, 2019) is only, and just only, for coding in R the Statistics you will learn through in (Salkind & Frey, 2020).
(Salkind & Shaw, 2019) it has the same content as (Salkind & Frey, 2020) but teaching it directly through R. I haven’t read it but it has probably everything, the Statistics and the programming in only one place. So it is worth the reading.

I highly recommend going through the first book (and probably the third too) because it is a game-changer about Statistics. But it takes some time. I spent on average 3 hours per day in each chapter.

The first two books, you can find “unauthorized” copies on the internet. The third one, I tried but I did not find it.

And feel free to shoot any questions ok?? I am more than happy to help if I can!!

#5

Thanks, lots of useful information there,
Best regards,
Anthony

#6

@diegoneto I’m going to be pestering you soon! I’m taking on-board your advice and I’ll get back when I hear from my module tutor.

#7

OK, I couldn’t wait! @diegoneto I re-read your post. The measure I was thinking about didn’t really stand up to much examination. So that leaves examining a relationship between two elements. I could do this, but I agree it doesn’t ‘feel’ as interesting as producing something new like an artefact.

Here’s a new slant I’d appreciate your point of view on. I’ve come across an article written by a professor in cybersecurity in Costa Rica and he has produced an architecture-based security conceptual framework to implement a security process. I would like to do some sort of comparison between this and ISO27001. ISO27001 is a management system which covers much more than just identifying threats and providing controls, but the output of Ricardo’s work looks similar to the risk assessment part of ISO/IEC 27001.

In particular I’d also like to look at both of these in the context of Industry 4.0 (where IT and operations are becoming more inter-linked). This would then be in the area of how both approaches deal with suppliers/external third parties.

However my problem is that I don’t know what I can produce. Given the difficulty of obtaining data around this area, what options do I have? Can I produce an imaginary example and work through how both approaches deal with the case? I’ve asked my course tutor but I’m also asking the cyber module author what he thinks too. I’d be interested to hear what you think and if it is possible to do a comparison, does this need an evaluation as well (apologies for the ‘basic’ questions here, research methods is really not my thing).

1 Like
#8

@SpanishTony I gotta say that when I read your post I thought “I will answer tomorrow” but the answers were already going through my mind and I said, “Oh fu$k it!! I’m gonna answer now.” :joy:

Considering you probably haven’t seen anything related to Statistics in your master’s, I believe you are making the right decision. I do not know how much time you have available for the course, and that it is an online one, so I believe keeping it simple and well done is your best strategy.

First, let me detail my experience so you can have a better picture of where I am coming from. For my master’s I did not write a thesis. I worked on a capstone project presenting a solution to a client. So from that standpoint, I may not have the best experience to share. But I recently dropped out from a Ph.D. program, I am applying for another two, and have been studying about the field this year. So my experience with research comes mostly from that background. Moreover, talking with my colleagues from that first Ph.D. program, many of them received feedback that their thesis could be rewritten/adapted to become one or two articles. So I am considering all documents to follow the same procedures and concerns.

With that being said, one thing to keep in mind is that, at least in the management field, every study you develop must bring novelty. You must find something new to work on. And keep in mind that it doesn’t need to be a word-changing novelty. You can focus on a small incremental one. And that would probably be better for your own control over the project. I am saying this because it sort of freaked me out when I heard the novelty thing was mandatory all the time.

And as I said, this is for the Management field. I can not say for sure your field is the same but if it is to take a wild guess, I would say that it probably is.

About the imaginary example, I would say that it is probably very difficult for this to be accepted because you would compare two existing models with no empirical data to analyze. What would be the data for these two models? Because, for example, if you had data from the two models for the same situation, you could assess how equal/different they are and explain what could be improved. But I believe the real data is paramount to comparing both frameworks.

And rest assured you are not alone in the quest for data because that is the major problem in research in social science. And considering nowadays Management research stopped creating value for organizations, the same organizations stopped sharing their data for research. So data became a very rare delicacy.

One question just ran through my mind. You mentioned the conceptual framework and the ISO27001 risk assessment part. How is the risk assessment measured? Is there a measure for that or is it just a walkthrough process that companies implement? Considering this is a process, I assume that any kind of measure, in this case, would be sort of a checklist right?

One thing that I thought when you mentioned about industry 4.0, do you see that those existing frameworks fall short on some aspects of industry 4.0? Like, they could somehow be improved and fit better this scenario? Because, and I do not know if that would be good for a thesis project, that would work as a theoretical paper considering all your proposals came through existing theory OR you could propose something, gather data through that proposal, and create an empirical study.

One thing that took me some time to absorb is that although it is recommended to have a measure validated through experts in the field (content validity) AND have it supported by existing theory and have it positively and negatively related to other constructs (construct validity), you can focus on only one of them when you create something. It is not THE BEST approach but sometimes you need to focus on what you can do with what you have. What I want to say is that if you, for example, (1) design a new framework more specific for industry 4.0 and (2) design a measure for such framework (reminding that I do not know if that measure is possible), you can measure other 2 or 3 constructs and focus on Construct Validity and “forget” Content Validity.

I hope I have helped more than getting things more confused for you. But one thing I would recommend you keep in mind is that the example situation you mentioned is just to for helping explain to others something you developed. Something you created. And that something must be supported by existing theory.

And keep thinking man! You will definitely go back and forth on this. I am more than happy to help you if I can. And feel free to ask all the “basic” questions you want to!! You must first cover the basics first so then you get into the more advanced stuff.

#9

Thanks for replying, your posts are really helping much more than the tutors on this course!

Perhaps just for context, the MSc I’m doing is online and the research module is two regular modules together (so 16 weeks - so not a PhD by any stretch). The current module is teaching us about putting together a research proposal. In theory, we have 4 weeks to put together a proposal in the research project module, but if you can nail it in this module then this gives us more project time in the 16 weeks.

Taking on board what you’ve said, I’ll change my approach. I think that there is an artefact that I can produce. A recent and comprehensive literature review around Information Security Management Systems (essentially what ISO/IEC 27001 encompasses) showed that there was a lack of research in InfoSec between companies, especially now in the time of Industry 4.0.

At a high level, what I’d be proposing then is a process/methodology that an organisation can request of another organisation around the controls it has in place around any element. These controls can be compared with the requesting organisation’s controls to see how much overlap there is. The advantage for the requesting organisation is that it can see how well their suppliers/customers might be handling their data, it’s also standard agnostic, so smaller companies can still show that they can be trusted with data (because they have adequate controls in place) even though they don’t have expensive 27001 certification.

At a lower level, how would this work? Still not sure, maybe some JSON API specification to request and receive the controls. Then some way to compare them with the requesting organisation’s controls and arrive at a score to say whether the controls match (or don’t). This could be implemented through VBA in an excel sheet or python. The difficulty is that the controls will be developed by different organisations and so there could be different words used for the same thing.

OK, but the low-level details part are details, whether something really basic gets developed or something mega using AI and machine learning matters not I think. What I’d like to understand is whether this is something that meets the minimum requirements for a proposal?

From your post, you can see that I am developing a framework (if that’s what the above idea is) and the framework will work with controls from existing theory (ISO27001, COBIT 5, Ricardo’s framework – do these count as measures?). Will this mean that I’ve enough to not worry so much about “Content Validity” and focus on “Construct validity”?

As I write this, I’m hopeful that what I’m proposing seems to make some more sense to me as a potential research project. I think I’m understanding that whilst the project itself might not move the world, the proposal/project must take a particular shape. However, if you can see some holes or things that I should have considered then please speak up.

Thanks,
Anthony

#10

@SpanishTony I’m glad to help!! Truth be told, this has also been helpful for me because explaining those things to you has been also a way to organize my thoughts about what I have been reading!!

I went through a similar process for my capstone project in my MSc. It was an accelerated program, the course was supposed to be hybrid but because of the pandemic it went all online. In addition, the instructor ended up being the project manager for the capstone. So she was more carrying us than giving us time to think about what was going to be our next action. If I’m going to be honest, I did not like my process because there were a lot of things to learn, to do and no time to reflect over them. But in the end, it is what it is.

Ok!! Now we are talking!!

I believe this is the thick of your research proposal. I am no expert in your field but with this text, you convinced me to pay attention to you because you will show me something relevant and new.

I agree with you that you should not be concerned about lower-level implementation because that can be done in a thousand different ways through different programming languages.

Yes, considering you are going to propose something new based on existing theory, I believe you are on the right path. Personally, I would go for more references than the 3 you listed but that is a matter of balancing time and scope. However, as far as I understand, I do not believe you need to be concerned about the content or construct validity. How are you going to consider the results of this workflow? Please correct me if I am wrong, but the answers you are going to get along your framework are going to be more qualitative rather than quantitative right? If you want to go beyond the approved/not approved in this control check to implement the communication, you could design some levels like red, yellow, and green (again, based on theory) labeling the degree of trust of the relationship. But that is just me spitballing.

Considering this project, I do not see any necessity (or even opportunity) to design any measure. Just a brief explanation about measures, in the Management field quantitative measures are basically surveys that people answer about their leadership, motivation, emotions, etc. Just like the picture below. And people would answer them from a range of 1 to 5 or 1 to 7.

image

Individuals answer two or three different surveys and then the researcher find the relationship between the constructs that the surveys were designed to assess. Moving away from surveys but still in the quantitative domain, I have seen some papers recording interviews or interactions and counting specific words related to the research topic and finding the relationship with other concepts. Taking a different path, if you wish to go for the story behind the facts you can go for the qualitative branch and do interviews or observe individuals in their environment to see where the comments/observations are converging.

So, in summary, if you go with this workflow (which I believe it is a good idea), I do not see any sort of opportunity, or even necessity, to have a measure related to that workflow. The only thing I can see, as I mentioned before, is some sort of categorization of the connection/data responsibility once the connection establishment process is finished. And that would show what kind of activities or data one company could trust the other with. But please take this with a grain of salt because I am just building those things as I go and from an outsider’s perspective.

Just remembering that this is the general perspective from the Management and Organizational Development field. I do not recall anything in these fields that is remotely related to a workflow just like you mentioned so I believe this is one of the topics that our fields differ. One thing I believe you could do is use the review paper you have as a starting point to search for, I don’t know, 3 or 5 papers that present similar workflows and see how, and even if, they measure their results. I do not know how experienced you are with reading academic papers (I wasn’t so that’s why I am explaining), but you do not need to read the introduction and literature review of an empirical paper and go directly to the methods procedures where the authors will explain how they measured the data they used in their work.

I honestly think you are on the right path. You are proposing something new, and you are basing it on existing theory. Those are the main topics for a research proposal. And you do not need to change the world with this. If you are able to change the conversation just a little bit, you are on the right path. And I believe you are on the right path. :+1:

As always, feel free to keep the questions coming!!

#11

Thanks Diego, you’ve been a big help. The course supplies a good template of what needs to be supplied to make a proposal and all these false starts have given me a good base of literature to reference!

What you have given me is an understanding of the reasoning for the path to take.

The risk assessment comes out with something similar, a 5x5 colour coded matrix, so it would be ironic if I could use the same tool to evaluate the framework.

I’ll speak to the tutor about the measure and see if he says. If he says it needs an evaluation then I can develop a simple enough survey or interview the one person I know who works in this area.

thanks again Diego, you’ve been a big help.

Anthony

1 Like
#12

Diego, cé é brasileiro?

#13

@SpanishTony I am more than glad to help!!

I do not see why not build upon this 5x5 matrix for your framework!! You will probably have to tweak a couple of things here and there because you are about to create your own framework. But considering there is already a peer-reviewed way to do that, there is no need for you to reinvent the wheel.

Yep. I’m on the fence about being proud of it :thinking: but yes. I’m from Rio.